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FOREWORD 


This report was prepared by the Man Systems Division of the URS/Matrix 
Company under Contract No. NAS8-25627^ Modification No. 15, "An Evaluation 
of the ATM Man/Machine Interface - Phase 3, Analysis of SL-3 and 4 Data", for 
the National Aeronautics and Space Administration, George C. Marshall Space 
Flight Center. The technical direction was provided by Mr. Lloyd B. Gardner 
(COR), Man/Systems Integration Branch, ELI 5. This report is the summary of 
the technical effort extending from May 24, 1974, to December 15, 1974. These 
contract results are documented In two volumes: Vol. I, the Executive Summary, 
and Vol. II, this Technical Report. 
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SECTION 1.0 




INTRODUCTION 


Presented in this report are discussions of the approach, the results and the 
conclusions of a study designed to assess past operations of a control console for a 
complex on-orbit manned system, with the expressed goal of applying these findings 
to the design and development of future on-orbit control and display systems. The 
report documents the activities and findings of Phase 3 of an overall study program 
to evaluate the effectiveness of the Skylab/ATM control/display system. 

1.1 SKYLAB/ATM BACKGROUND 

Launched In 1973, Sky lab was the United States' first manned space station. 
This vehicle was capable of supporting prolonged manned missions with program 
objectives related to study of the earth, the sun, man and space technology. The 
station was manned for a total of 171 days in three periods (SL-2, 28 days; SL-3, 
59 days; and SL-4, 84 days) , 

On Sky lab, the sun was the subject of intensive investigation by a manned 
solar observatory — the Apollo Telescope Mount (ATM) . The ATM consisted of 
an integrated set of eight telescopes to observe, monitor and record the structure 
and behavior of the sun and its corona. It also included the necessary guidance 
and navigation systems for attitude control and telescope alignment, thermal 

conditioning systems and electrical power systems. 
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Of the total man-hours spent on Skylab (11,918)^ approximately 8% of the 
time was spent in solar observations. Instead of the 880 planned man-hours of 
investigation, the crews actually spent 941, as shown In the table below (this 
excludes 156 man-hours spent on SL-4 for the Comet Kohoutek) , 


Table 1-1: Solar Observation Time Per Manned Period 


MANNED 

MAN-HOURS 

% OF 

PERIOD 

UTILIZED 

TOTAL 

SL-2 

117 

6.0 

SL-3 

305 

7.8 

SL-4 

519 

8.5 

TOTAL 

941 



Of the nearly 12,000 man-hours of orbital time, over 3,000 were spent In scien- 
tific investigation, with approximately 940 man-hours (or 30%) devoted to solar 
observations. Overall scientific investigation time is summarized in the following 


table. 


Table 1-2: Scientific Investigation Per Manned Period 


MANNED 


SOLAR 

EARTH 

CORRELARY, 
STUDENT & 
COMET 

TOTAL 

SOLAR 

OBS. 

% OF TOTAL 

PERIOD 

MEDICAL 

OBS. 

RESOURCES 

KOHOUTEK 

PERIOD 

PERIOD 

SL-2 

145 

117 

71 

65 

398 

29 

SL-3 

312 

305 

224 

244 

1085 

28 

SL-4 

337 

419 

274 

503* 

1633 

32 

TOTALS 

794 

941 

569 

812 

3116 

30 


^Includes 156 man-hours spent in Comet Kohoutek using ATM equipment. 
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1.1.1 ATM Configuration 

The ATM cluster consisted of the fohowlng instruments covering wave- 
lengths ranging from 2 - 6,500 Angstroms (including two targeting telescopes 
operating in the Hydrogen-Alpha wavelength) : 

• Hal and 2 Telescope 

• White Light Coronagraph (S052) 

• X-Ray Spectrographic Telescope (S054) 

• UV Scanning Polychromator/Spectroheliometer (S055A) 

• X-Ray Telescope (S056) 

• XUV Coronal Spectroheliograph (S082A) 

• UV Spectrograph (S082B) 

These instruments sensed various solar phenomena and converted them to 
solar images, line spectra and photometries, which were converted to a video 
presentation, recorded on film, recorded on tape, or telemetered to the ground. 

The ATM was mounted on the orbital assembly such that both it and its 


solar panels were directed toward the sun during solar observations. The 
control console for this observatory cluster was located in the Multiple Docking 
Adapter (MDA) , a cylinder 10 feet in diameter and 17 feet long, mounted to the 
Airlock Module (AM) above the Orbital Assembly (OA) . In addition to the ATM 
control crew station, the MDA also housed the Earth Resources Experiment 
Package (EREP) , the Materials Processing Facility, and miscellaneous experiment 
storage. The console was located just forward of the MDA/AM interface. 


1.1.2 ATM Control /Display Console 

This integrated control /display console was originally designed for seated 
operation In the Lunar Module (LM) In an earlier Skylab cluster configuration. 
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Later in the program, however, it was relocated to the MDA, and the crew 
elected to operate It from a standing position. This crew station consisted of 
the ATM Console, the Foot Restraint Platform (Astrogrid) , a Skylab Restraint 
Assembly (a chair, used only by the SL-2 crew) and a Speaker Intercom Assembly 
(see Figure 1-1). The restraint platform was adjustable to accommodate variations 
In the total crew (5th - 95th percentile range) . 


Figure 1-1: Solar Observatory Console (ATM) Crew Station 
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As illustrated in Figure 1-2, this complex console contained the controls and 
displays for seven solar experiments, instrument cluster pointing control, elec- 
trical power monitoring and control, attitude and stability, two video monitors, 
etc. The console contained over 1700 square inches of panel space on two planes 
The experiment portion of the panel alone housed 96 switches and 37 displays 
of various types. Each experiment had scientific as well as hardware monitoring 
status parameters which were measured and telemetered to the ground stations. 
For example, S082A had 130; S054 had 68. These data were used as inputs to 
the study. 

In order to effectively and efficiently use this complex console in an 
Integrated fashion, various procedure systems were developed called Joint 
Observing Programs (JOPs) . These JOPs (e.g.. Program No. 3 - Flares, and 
No. 14 - Solar Eclipse) enabled Instruments to collectively observe solar phenomena. 
The JOPs consisted of various combinations of intrument configurations, or “Building 
Blocks" (BBs), thus providing flexibility in selecting the data acquisition method 
for a particular solar investigation. The JOPs performed on orbit yeilded a volumin- 
ous amount of excellent telemetered and film data on solar phenomena. 

1.2 PREVIOUS CONTRACT EFFORTS 
1.2.1 Phase 1 Activities 

In preparation for the assessment of the ATM control /display station inter- 
face, the ATM control panel and panel operating procedures were examined to 
determine what data would be required to reconstruct the panel's operation, 
post-mission. The projected ATM telemetry measures were then examined to 
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determine whether the required measures were available, and, upon finding that 
they were, those that were appropriate to the study were requested. 

1-2.2 Phase 2 Activities 

In this phase, a compilation of baseline ground data suitable for statistical 
analysis, a preliminary analysis of SL-2 on-orbit data, and a preliminary statis- 
tical comparison of the baseline ground data and the on-orbit data were generated. 
The results obtained from comparisons of ground based and on-orbit data showed 
that the same types of errors made during one-g simulations were also made 
during the on-orbit panel operations. 

The findings of the Phase 2 report (Ref, Appendix A) , based on an analysis 
of the SL-2 ATM operations, were: 

• The left side of the panel had a higher deviation (error) rate than 
the right side . 

• Three-position toggle switches (F-F-F) used without a display had a 
high deviation rate. 

• Experiment modes that required manual timing by the crewmen were 
often timed improperly. 

• Many of the deviations which occurred during operation of S082B 
involved confusion with S082A ("B" was set up to "A" specifications).* 

• Building Blocks with several subsections separated by pointing commands 
had high deviation rates for the centermost subsections. 


*This was postulated to be due to similarities In controls/displays labeling 
nomenclature, and BB instructions. For SL-3/4, large red and blue decal 
letters "A" and "B" and red and blue boundaries (taped strips) were applied 
to experiments in an attempt to alleviate confusion and, thus, reduce error 
rates. 




• isolated control actuations had a much higher deviation rate than 
sequential control actuations. 


1.2.3 Applications to Future Missions 

Added to the Phase 2 effort was a task to develop a preliminary set of 
design recommendations, based on the analysis of the deviations in SL-2 panel 
operation . 

Since the operations performed on the ATM console are similar to those 
projected for the Space Shuttle payloads, the considerations derived from this 
effort should have direct application for the development of payload control 
station design criteria and guidelines. These design recommendations are sum- 
marized below: 

• The number of subsystems (experiments) controlled from the panel's 
left side should be no greater than the number of subsystems controlled 
from the right side, assuming similar control /display requirements 
(control and display density) . 

• The number of time-phased subsections of Building Blocks should be 
minimized (preferably four or less) . 

• Three-position toggle switches (F-F-F) should have a visual display 
for position Indication. 

• Experiments that require timing of experiment modes should have 
dedicated integral timers. 

• Isolated control actuations should be minimized. If necessary, they 
should be accompanied by visual or audible displays, or should be 
clearly defined in the panel operation procedure. 

• If several subsystems (experiments) must be controlled from one panel, 
similar experiment identification nomenclature (e.g., S082A, S082B) 
should be avoided. Similar nomenclature for individual controls or 
displays should also be avoided to prevent confusion, especially when 
the subsystems are located near each other on the panel . 
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These data were subsequentfy incorporated into MSFC-STD 512, Man/ 
System Design Criteria for Manned Orbiting Payloads. 

1.3 ANALYSIS OF SL-3 AND SL-4 MISSION DATA (PHASE 3) 

Although collection, reduction and analysis of SL-2 ATM panel operations 
in Phase 2 produced preliminary design criteria applicable to future missions, 
many of the panel operational problems could not be correlated with panel 
design features due to an insufficient quantity of data samples (3 crewmen^ 
shortest SL manned mission length) . To provide answers to many questions 
left open by the SL-2 effort, and to increase the con fid enjce level of the resulting 

I 

design recommendations, an additional effort to collect, reduce and analyze the 
SL-3 and SL-4 data was undertaken. 



The overall goal for the Phase 3 effort and, indeed, for the entire ATM Assess- 

\ 

ment Program, is to update and improve human factors hardware design and proce- 
dural guidelines, in order to contribute to improved future manned systems perform- 
ance. The major objectives outlined and achieved toward that end,dun‘i^g Phase 3, 
were: 

1) Completion of a comparison of Phase 2 results (SL-2 data) with results 
obtained from the Phase 3 data analyses. 

2) Identification of hardware design and procedural parameters contributing 
to unacceptable ATM experiment system performance. 

3) Development and presentation of design guidelines and recommendations 
for overcoming the unacceptable performance identified in 2) above. 
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) Development of a concept and construction of a soft mockup of an optimized, 
modular control panel envelope capable of accommodating either one or two 
crew members using ATM component configurations, and being integrated 
mto future Space Shuttle payloads (e.g.. Payload Specialist Station) . 

In the original contract Scope of Work, results of ATM Assessment were to be 
compared against future mission requirements. The outcome of that comparison was 
to include identification and documentation of payload missions which would benefit 
from the criteria and recommendations ensuing from the assessment study and presen- 
tation of that information to payload mission designers as an aid to improve the man/ 
system Interface. However, in lieu of performing that comparison, the contracting 
officer directed that a statistical analysis of the data be performed. In accordance 
with his direction, a statistical analysis was incorporated Into the study and the 
comparison dropped. 

) 

Description of SL-3 and SL-4 Crew and Crew Training 

As in the previous manned mission (SL-2), the station was manned by three 

/ 

crew members. Each mission included a Commander, a Scientist Pilot, and a 
Pilot. 

Although the scientist pilot (as his name implies) was responsible for the 
scientific aspect of the mission, all of the crewmen were cross-trained and, 
therefore', capable of operating the ATM console. 

The crews were exposed to various levels and complexity of training for 
their respective missions, as shown in Tables 1-3 and 1-4. Although the crew 
for SL-4 had more time in which to prepare, due to their flight date being the 
latest, they also had less time between the final stages of JOP development and 
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flight date. An attempt was made to keep the workload for SL-U the same as for 
SL-3, but, with the added complexity of the mission (e.g., addition of investiga- 


tion of the comet Kohoutek) , the overall time between on-orbit operations was less, 
thus creating greater ATM activity density for the SL-^ crew. 


Table 1-3: Types and Length (in hours) of 

Astronaut Training - Total ATM 


TYPE 

SL Simulator 
Solar Physics Briefings 
Experiment Briefings 
MSFC Simulator 


SL-2 

SL-3 

SL-4 

286 

419 

431 

290 

338 

496 

82 

’ 180 

160 

26 

28.5 

65 


Table 1-4: 

Cha racteri sties 

of JOP 


CHARACTERISTICS 

SL-2 

SL-3 

SL-4 

Original No. of JOPs 

13 

15 

23 

No. of Complex JOF^s 

j 

8 

13 

18 

No. of Late Modifications 

1 

3 

2 

8 

Level of Late Modifi^cations 

- 

Moderately 

Heavy 

Very 

Heavy 

No. of "Shoppting List" 

0 

2 

2 

Time Between JOP Dev. and 

3 

U - 2 

<1 


End of Training (months) 



1.4 PHASE 3 - REPORT OVERVIEW 


The Phase 3 study delved Into two independent data collections: 1) telemetry 

data, and 2) voice transcript data. Statistical information about panel usage was 
derived from telemetry. Due to the voluminous nature of the SL-3 and SL-4 
telemetry data, a fixed interval sampling technique was employed in the selection 
of mission time periods to be analyzed. The voice transcripts were reviewed to 
document operational problems. In the analyses performed on those data, the 
following types of factors were addressed in order to assess their effect on ATM 
panel man/machine Integration: 

• Control and display component type 

• Control and display location 

• Operating procedures 

• Time of operation in mission 

• Panel layout 

The methodology used to extract, analyze and test the compiled data is 
documented in Section 2.0. The data analyses, results and conclusions for the 
two data collections, as well as the results and conclusions of the hypothesis testing 
are given in Section 3.0. Section 4.0 contains integrated study results, conclusions 
and recommendations for future work. 
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SECTION 2.0 


METHODOLOGY 

2.1 GENERAL APPROACH 

The general approach taken In assessing the man/machlne interface on SL-3 
and SL“M incorporated separate analyses of two types of data: .1) telemetry data, 
and 2) critical incidents noted in the missions' voice transcripts. (See Figure 2-1, 
Overall Task Flow.) This double approach was used to obtain the most realistic 
representation possible of the ATM man/machine system. 

The telemetry data analysis employed a fixed-interval sampling technique 
to review routine ATM operations. The critical Incidents analysis augmented this 
routine sampling with information derived from comments made by the flight crews 
and ground support during and after periods of ATM operation. Subsequent to 
separate reviews of the two bodies of amassed data, comparisons were made in 
order to reveal areas of agreement or conflict. Each such area was discussed, 
and the resulting conclusions were reported. 

The overall result of the effort is a set of design recommendations suitable 
for incorporation into future mission design. The discussion that follows describes 
each major phase of the study, the assumptions made, the methods employed, and 
the limitations established. 
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FIGURE 2-1: Study Task Flow Diagram 





2.2 TELEMETRY DATA ANALYSIS 
2.2.1 Data Point Selection 

In order to identify periods of manned ATM experiment operation, the 
Skylab Mission Events document (Ref. Appendix A) was reviewed. This docu- 
ment describes the activities of each crew member throughout the Skylab 3 and 4 
missions. After mission time and duration of each manned ATM operation were 
identified, the data tapes containing the associated telemetry were located and 
organized. 

A computer program developed by NASA/MSFC during Phase 2 of this con- 
tract was used to extract relevant data from the telemetry tapes. Certain infor- 
mation available through telemetry channels appeared to contain useful measures 

of man/machine performance. A list of the readouts for each ATM experiment is 

/ 

given in Table 2-1. The distribution of components corresponding to each readout 
and the layout of the ATM control and display panel, as it was flown on SL-3 and 
SL-4, is Illustrated In Figure 2-2. 

The ATM panel was selected for analysis for several reasons: 

1 . The panel represents the most sophisticated, integrated experiment 
; control panel ever used in manned space flight. 

I 

2. A considerable amount of data were telemetered to the ground to 
assist ground-based scientists in planning observations. 

3. The panel has undergone some man/machine study during the develop- 
ment and verification process via man-in-the-Ioop simulation. Results 
of these ground-based simulations were compared with on-orbit results 
in an earlier study phase. 

4. Some components of the panel represented novel applications of 
previously used components, and some were entirely new components. 
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Table 2-1; List of Readouts for Each ATM Experiment 


Ha FR/MIN, 3-POS TOGGLE SW (F-F-F) 

START-STOP, 3-POS TOGGLE SW (F-F-F) 

S056 MODE, ROATRY SW, 11-POS 

EXPSOURE SW, 3-POS TOGGLE (F-F-F) 

FLARE ENABLE, 2-POS TOGGLE (F-F) 
START-STOP, 3-POS TOGGLE (M-F-M) 

S082A MODE, TOGGLE SW, 3-POS (F-F-F) 

WAVELENGTH TOGGLE SW, 2-POS (M-F-M) 
EXPOSURE TOGGLE SW, 2-POS (M-F-M) 
START-STOP TOGGLE SW, 2-POS (M-F-M) 

FLARE ENABLE TOGGLE SW, 2-POS (M-F-M) 

S082B MODE, TOGGLE SW, 3-POS (F-F-F) 

WAVELENGTH TOGGLE SW, 2-POS (M-F-M) 
EXPOSURE TOGGLE SW, 2-POS (M-F-M) 
START-STOP TOGGLE SW,‘ 2-POS (M-F-M) 

FLARE ENABLE TOGGLE SW, 2-POS (M-F-M) 

AUTO SEQ TOGGLE SW, 2-POS (F-F) 

S052 MODE, ROTARY SW, 5-POS 

START-STOP TOGGLE SW, 3-POS (M-F-M) 

MIRROR POSITION TOGGLE SW, 3-POS (M-F-M) 

S055A MODE, 7 DETECTORS, 3-POS TOGGLE SW (F-F-F) 

ROTARY SW, 9 POS 

START-STOP TOGGLE SW, 3-POS (M-F-M) 
GRATING, 2-POS TOGGLE SW (F-F) 

GRATING POSITION INDICATOR 

S054 CRATING, 3-POS TOGGLE SW (F-F-F) 

FLARE ENABLE, 2-POS TOGGLE SW (F-F) 
START-STOP, 3-POS TOGGLE SW (M-F-M) 
PICTURE RATE ROTARY SW, 4-POS 
EXPOSURE RANGE ROTARY SW, 6-POS 
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The particular components chosen for print-out from the telemetry data 
tapes were selected to meet certain criteria. Those criteria are listed here in 
order of priority, with most important first: 

1. The component was monitored and telemetered to the ground. 

2. The component was involved in key portions of experiment operations. 

3. The component complemented other telemetered components in distribut- 
ing the selected hardware items throughout the experiment portion of 
the control /display panel. 

4. Data from the component could be used as a performance measure when 
compared to established procedures. 

5. The component complemented other components in distributing selected 
hardware items across types of C/D hardware available on the panel. 

From the tapes, data for selected periods of interest in SL-3 and SL-4 
were printed out. By sampling every sixth building block from the PAAS 
(Ref, Appendix A) document, the objective of sampling at fixed intervals throughout 
the missions was satisfied. This procedure provided 273 total data points to be 
analyzed. A sample sheet from the PAAS is shown in Figure 2-3 with relevant 
operations marked. 

A sample print-out sheet for one of the experiments Is presented in 
Figure 2-4. Referring to the figure, it can be seen that the print-out provides 
accurate data on mission time for each operation related to that experiment. 

2.2.2 Data Point Scoring 

Scoring the telemetry data involved development of a procedure which reflected 
awareness of several facts regarding the nature of the information. 
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Figure 2-3: Sample Page from PAAS Document 
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2.2.2. 1 Data Limitations 


The procedure used in scoring the telemetry data (Figure 2^5) Included a 
review of the computer print-outs, the Building Blocks, the voice transcripts, the 
PAAS document, and the Mission Requirements Document (MRD) for each data point. 
One of the major shortcomings of a retrospective analysis In an operational environ- 
ment (as opposed to an experimental environment^ is the inability to control variables 
This problem manifested itself in the form of unwritten or unrecorded instructions. 
The analysis reported here is based strictly on performance of the ATM system as 
compared to updated Building Block instructions. Any "understood" procedures 
which were not recorded In the mission documentation were not reflected in the 
analysis. Consequently, if it was understood by the crew that operational times 
stated for experiments were minimum times, not time limits, overrunning specified 
times would be appropriate, while underunning would not be appropriate. In this 
analysis, however, underrunning and overrunning were evaluated equally because 
there were no stated instructions to indicate that missing the stated time was accept- 
able. 

Another case In which the problem of unrecorded instructions could have 
prevailed is that of "understood" procedural deviations. For example, the S055A 
experiment has a mode switch which allows three mirror raster scans or continuous 
(auto) raster scans. An understood procedure may have applied in which three 
rasters at any selected site were adequate. Consequently, by selecting the three- 
raster mode when formal instructions dictated selection of auto raster, the crewman 
avoided having to manually terminate the mode. This could have afforded additional 
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STATISTICAL ANALYSIS DATA POINT SCORING PROCEDURE 

1) Secut*e data point computer print-out, 

2) Note mission time on print-out (ref. Step 1) . 

3} Secure voice transcripts covering data point performance times (-t- 1 hr.; 
ref. Step 2) . 

4) Secure PAAS document. 

5) Secure MRO for appropriate mission (SL3 or SL4) 

6) Secure ATM experiments description summary document. 

7) Secure ATM C&D panel schematic. 

8) Secure 1 ea. raw data score sheet. 

9} Refer to data point print-out (ref. Step 1) and complete Information across 
top of score sheet (ref. Step 8; e.g., data pt # JOP # , etc.) , 

10} Determine data point BB number (ref. Step 9] . 

11) Find BB number (ref. Step 10] bar chart In MRD document (ref. Step 5) . 

12) Note data point performance time (ref. Step 9} . 

13) Refer to PAAS document (ref. Step 4) , Find times that bracket performance 
period (ref. Step 12) and that pertain to data point BB (ref, St^ 10) , 

14) Record on a note pad special conditions (as presented In PAAS) for modifying 
BB ordered activity (ref. Step 11) , 


15) Determine error opportunities; 

15a for each ATM experiment In BB (ref.Stepll) note on a pad the control 
or display activity that must be performed. 

15b modify BB steps (ref. Step 15c) as per PAAS Instructions (ref. Step 14) . 

ISc refer to voice transcripts (ref. Step 3) . 

ISd update modified BB activity (ref. Step tSfo) as per voice transcripts. 

ISe refer to computer print-out (ref. Step 1) — modified and updated B8 

activities (ref. Step I5d) that can be checked via telemetry data for 
occurrence of errors will be considered error opportunities. ‘ 

16) Enter error opportunities (ref. Step 15e) on score sheet (ref. Step 8) . 

16a for each control and display error opportunity (ref. Step 15e), check 

ATM C6D panel schematic (ref. Step 7) to determine associated hardware 
type. 

16b enter one mark for each error opportunity In appropriate score sheet cell. , 

16c while performing Step 16b, nolo whether error opportunity Is sequential 
or single operation and place dot In appropriate score sheet cell. 

17) Score Errors 

17a From data print-oul (ref. Step 1), check data point time of performance. 

17b Review the prlnt-out data for the time of performance (ref. Step 17a) 

and determine which control or display error opportunities (ref. Step 15e) 
actually resulted In errors. 

17c Determine error type and enter first letter of that type In appropriate 
score sheet cell , 

17d While performing Step 17c, note whedier error is sequential or single 
generation and enter dot In appropriate score sheet cell. 


Figure 2-5; Procedure for Scoring Telemetry Date 
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time for the crew to conduct other experiments without interruption. 

As the data points were selected, it became apparent that the data tapes 
available included only afternoon and evening operations {GMT 1200-2400 hours) ; 
A concerted effort was made to obtain morning tapes, but it was discovered that, in 
most instances, they were not even existent, due to limitations in ground taping 
capacity during actual mission flight. After the missions, although the data for most 
of the morning hours had been telemetered to the ground, MSFC's Skylab Office com- 
puter time limitations did not allow transference of the Information onto tapes which 
could be programmed to print out ATM operations. The morning time periods were, 
therefore, eliminated from the analysis. 

2. 2. 2. 2 Distribution of ATM Operations Intervals Analyzed 

The fixed-interval sampling technique encompassed data points ranging 
across most of SL-3 and SL-4 orbital time. Mission times from Day 219 to 
Day 264 In SL-3 and from Day 331 to Day 398 in SL-4 were sampled. Of the 
28 Building Blocks available for use in the SL-3, a total of 14 were exposed to 
at least one analysis as a data point. Of 39 Building Blocks available for use 
in SL-4, 19 were subjected to analysis. This distribution of data points across 
Building Blocks is summarized in Table 2-2. 

2. 2. 2. 3 Building Blocks and Other Determinants of Scheduled Operations 

The Building Blocks (BBs) were considered the primary procedures instru- 
ments for Skylab ATM operations. A sample of these formats, BB #28, is presented 
in Figure 2-6. As can be seen in the figure, the mode, exposure settings, start 
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TABLE 2-2; Distribution of Building Blocks 
Across Telemetry Data Analysis Data Points 


SL-3 


SL-4 


BUILDING 

NUMBER OF 

BUILDING 

BLOCK 

DATA POINTS 

BLOCK" 

NUMBER 

ANALYZED 

NUMBER 

10 

28 

10 

2 

20 

1 

1 

7 

28 

11 

7 

32 

4 

5 

2 

36 

5 

4 

5 
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Figure 2-6; Sample Building Block (BB #28J 
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time, duration, number of frames of data taken, etc. , is shown for each experiment 
operated in a selected Building Block. The clear or unshaded areas represent 
periods of operation for each designated experiment. 

Instructions given via the on-board teleprinter were recorded In the PAAS , 
and were considered to supersede the Building Block procedures. Similarly, 
verbal instructions discovered in the voice transcripts were considered to super- 
sede any other instructions. Therefore, procedures indicated in the Building 
Blocks as modified by the PAAS and/or the voice transcripts, were used as the 
reference for intended operations. Actual experiment operations (as indicated 
in the telemetry data print-outs) were compared with the expected operations 
to identify errors or deviations. Each data point (total of 273) was scored 
according to the procedure outlined In Figure 2-5, above. A sample Raw Data 
Score Sheet used for reporting results on each data point is presented in 
Figure 2.7. 

2. 2. 2. 4 Implementation of Data Scoring 

One of the difficulties encountered in any analytical process is appropriately 
defining ground rules, assumptions, terms, etc., relating to the analysis. It Is 
essential to the process to have clearly defined terms and supporting assumptions 
for use by each member of an analytical team. Therefore, before implementation 
of data scoring, operational definitions were developed for use by the analysts, 
in order to assure uniformity of data interpretation. These operational defini- 
tions, and several assumptions which were made prior to data analysis, are 
presented in Figure 2-8. 
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Figure 2-7: Sample Raw Data Score Sheet 














Operational Definitions: 


Control Hardware item that functions primarily to initiate a change of 
status. 

2) Display - Hardware item that functions primarily to provide operator with 
information. 

3) Control Error - Incorrect control positioning or incorrect control sequenc~ 
ing. 

Display Error - Inappropriate performance of an operation dependent upon 
information feedback apart from that provided by a control. 

5) * Control Error Opportunity - Control action required by amended Building 

Block instructions. 

6) * Display Error Opportunity ~ Building Block ordered activity dependent 

upon information feedback divorced from that provided by controls. 

■f 

7) Single Operation - An operation separated from its related operations by 
more than three other events or by more than one minute. 

Assumptions : 

1) initiate or terminate actions will be scored as display error opportunities 
or as display errors when they are keyed to time or unrelated event cues. 

2) The ATM Experiments Description Summary document will define equipment 
operating modes. 

3) An error opportunity exists only when sufficient telemetry data are provided 
for analysts to determine whether or not an error was committed. 


*Refer to Assumption number three. 


Figure 2 8: Statistical Analysis Operational 
Definitions and Assumptions 
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In exercising the analytical tools to develop forms, definitions and proce- 
dures, It was discovered that, based on telemetry data, it is difficult to dis- 
tinguish between control errors and display errors. For example, on the S055A 
experiment, instructions were given to the crewmen to set the grating at a 
pre-established reading. A digital readout was provided to indicate the grating 
position. If the operator set the grating at 0112 instead of the instructed 0157, 
is this the result of misreading the display or neglecting to initiate a stop 
command to stop the grating scan at the appropriate time? In cases such as this, 
the error was classified as a display error. By the operational definitions, the 
"inappropriate performance of an operation dependent upon information feedback 
apart from that provided by a control" is considered a display error. 

In the specific case of the S055A' grating position indicator, provisions 
were made to allow some latitude around the instructed value. A fine point 
ti.e., +0005) tolerance around the desired value was established as a reasonable 
band for positioning the grating. This limit wis based on the fact that the 
units digit of this display changed so rapidly that it was nearly impossible to 
exactly position the grating without single stepping. Consequently, the tens 
digit was very likely used for positioning the grating. 

Limits or tolerances also had to be established for time sequences. For 
example, if the Hal camera was intended to be operated at 2 frames/min. for 
3 minutes. How much time short of and beyond the two minute desired time 
was considered acceptable? A basic guideline of +10% of the desired time was 
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.considered acceptable. Excursions outside this +10% band were classified as 
time displacement errors. 

The Raw Data Score Sheet (Figure 2-7) provided space for recording 
reference Information on each data point. Across the top of the sheet are spaces 
for recording the number of the data point. Joint Observing Program (JOP) 
Number, Building Block (BB) number, and mission time. . This information was 
available in the PAAS document, with the exception of the data point number 
which was arbitrarily assigned by URS/Matrix analysts when the time periods 
for analysis were Initially selected from the PAAS. 

The remainder of the score sheet was devoted to man/machine performance 
scores for the various components monitored via telemetry for each ATM experi- 
ment. The column at the far left of the score sheet included the designators 
for the ATM experiments. The scientific name corresponding to each of the 
designators is presented in Table 2-3. 

Table 2-3: ATM Experiments Evaluated 

Designator Code 

Hal and 2 
S052 
S054 
S055A 
S056 
S082A 
S082B 


Scientific Name 

Hydrogen Alpha #1 and #2 

White Light Coronagraph 

X-Ray Spectrographic Telescope 

UV Scanning Polychromotor/Spectroheliometer 

X-Ray Telescope 

XUV Coronal Spectrohellograph 

UV Spectrograph 
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The horizontal axis of the data score sheet included columns for control 
components and display components. The abbreviations after the title of each 
type of toggle switch refer to the characteristic of each switch position. These 
abbreviations are: F, fixed; M, momentary; L, latching; SL, spring-loaded; 

FC, fix-guarded. The display component types are listed in the adjoining 
columns. 

Space was provided at the right side of the data score sheet for classifying 
the activity as either a sequential or a single operation. The next set of columns 
afforded space for calssifying the error type. Errors of omission, inversion and 
time displacement were mutually exclusive categories. The extreme right-hand 
column provided space for remarks which were felt by the analyst to be relevant 

I 

to interpreting the score sheet entry. 

The cells In the data score sheet provided space for recording number of 
errors committed (as inferred from comparison of expected activity and telemetered 
readouts) and error opportunities. The quotient of these two quantities is then 
used as an error rate in subsequent calculations. In no case is it possible to have 
an error committed without an error opportunity. That Is, even though a haphazard 
event may have been recorded from telemetry which cannot be reconciled with 
procedural instructions. It Is not recorded as an error without an error opportunity. 

I 

In this case, an error opportunity was entered in the opportunity space to simplify 
calculations which were to follow. 

2.2.3 Inter-Rater Reliability 

The data point scoring was performed by three analysts. Prior to the actual 
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analysis process, these individuals, together with two other senior analysts con- 
ducted data scoring exercises with candidate operational definitions and candidate 
scoring sheets. As several data points were scored, areas of conflict were discussed 
and the operational definitions and score sheets were modified to alleviate problem 
areas. This procedure also served as an opportunity to "walk through" the scoring 
procedure for training purposes. Thus, inter-rater variability was reduced to a 
minimal level. Prior to the major data point scoring effort, a single data point was 
scored independently by each of the analysts. A comparison revealed no substantial 
differences In error rates derived by the three analysts. 

2.2.4 Statistical Analysis 

2.2.4. 1 Descriptive Statistics 

To report the results of the telemetry, data descriptive statistics were 
prepared summarizing the distribution of error rates across the reference 
variables. Those descriptive statistics included three methods of analysis: 

1) a cell to cell comparison of Table 3-1 data, 2) a look at parameter effects, 
and 3) an in-depth analysis of factors contributing to unacceptable error rates. 
Tabljes and discussions supportive to those analyses are provided in subsequent 
sections of this report to describe and compare error rates by experiment, 
mission time. Building Block, etc. 

2. 2. 4. 2 Hypothesis Testing 

As a result of a study of SL-2 telemetry data under an earlier contract phase 
and the preliminary findings resulting from that study, several hypotheses were 




developed for testing In the SL-3/SL-4 analysis. These hypotheses are: 

1 . The use of three-position toggle switches produces a significantly higher 
error rate than does the use of other types of toggle switches. 

2. The use of rotary switches with positions numbering more than four 
produces a significantly higher error rate than does the use of rotary 
switches with four positions. 

3. The use of rotary switches with positions numbering more than four 
produces a significantly higher error rate than does the use of other types 
of switches investigated. 

4. Experiments operated on the left side of the C&D panel produce a signifi- 
cantly higher error rate than do experiments operated on the right side. 

5. Operating Hal produces a significantly higher error rate than does 
operating any other experiment on the left side of the panel . 

6. Performing "isolated actuations" produces a higher error rate than does 
performing "sequential operations". 

7. There is no significant difference in performance between operating 
S082A and operating S082B. 

8. Operating experiments with high GSD layout similarity (e.g., Hal, S052, 
S082A, S082B) produces a higher error rate than does operating experiments 
with dissimilar C8D arrangements (e.g., S056, S055A, S054) . 

9. Operating the lower half of the ATM panel produces a significantly higher 
error rate than does operating the upper half of the panel. 

10. SL-4 operations produce lower error rates than do SL-3 operations. 

11 . The first one-third of each mission's operations produce higher error 
rates than do the remaining two-thirds of each mission's operations. 

Two statistical methods were employed to test the hypotheses. These were 
chi-square and a 2 x 2 analysis of variance for repeated measures. Results of the 
statistical analysis appear in Section 3.1.2. 







2,3 CRITICAL INCIDENTS ANALYSIS 
2.3.1 General Approach 

Because the telemetry data analysis employed a sampling strategy, it was 
felt that a review of all the available SL-3 and SL-4 voice transcripts could reveal 
other periods of problems that might be of interest and could provide a different slant 
on the operation of the ATM control /display panel. Consequently, these transcripts 
were obtained and reviewed to identify relevant remarks. 

2*3.2 Nature of the Data 

In reviewing the voice transcripts for the SL-3 and SL-4 missions. It was 
recognized that an information body very different from the telemetry data was 
involved. The most outstanding difference was the subjective nature of the informa- 
tion -- being composed of the actual statements and questions made by the flight 
and ground crews during and after the ATM passes. Rather than having a record 

of exactly what happened, the analysts had a partial record of what was perceived . 
That the voice transcripts contain only flight and ground crew perceptions is 
obvious -- something must be perceived before a comment about it may be voiced.* 
The "partial" characteristic of the data may be explained by four limitations: 

1) It is not reasonable to assume that everything perceived was voiced aloud. 

2) Not everything voiced aloud was recorded. 


*The exception to this is mission time, which was coded on the tapes as comments 
were recorded . 
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3) Not every recorded verbal comment was translated to tape. 

iO Not every taped comment was included in the edited transcriptions of 
the tapes. 

As URS/Matrix analysts worked from the MSFC/HOSC collection of edited voice 
transcripts in culling for critical incidents, they had to develop a methodology which 
would allow collection of as much information as possible related to the ATM man/ 
machine Interface While, at the same time, accommodating the inherent limitations 
of the data body . 

2.3.3 Types of Information Available 

Certain general information was available in the transcripts. Mission day 
and time were coded as comments were made. Occasionally, the flight crew or 
ground control referenced their comments to specific JOP, Building Block, Shopping 
List, and/or experiment numbers. The comments ranged from very broad remarks 
about whole panel layout and ATM pass scheduling to very specific comments or 

questions about number of seconds exposure for a certain frame in a certain experi- 
ment and problems encountered with a specific control or display. The analysts 
were able to determine from the comments, with few exceptions, whether the remark 
was of a positive or negative nature. Adding to these remarks the analysts' own 
knowledge of panel hardware and experiment procedures, they were able to infer 
a relative level of severity or benefit to overall ATM, specific experiment, or 
specific hardware item function intimated by the remark. 
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Qt^ganlzation and Reduction of Voice Transcript Derived Data 
To organize all this information, a Critical Incidents Raw Data Score Sheet 
(Figure 2"9) was generated. This form enabled the analysts to record as much 
information as was available from the voice transcripts (not every blank was filled 
in on every sheet) . This information was further reduced by tallying and subsequent 
across-hardware type, across-procedure, across-experiment, and across-mission 
comparison, with the experiments rank ordered by frequency of remark occurrence. 
The comparison by remark frequency was then complemented by a study of the 
relative impact of a noted problem, regardless of the number of times it was mentioned. 
This was done to allow for those Instances in which a small number or single remark 
may have revealed an important problem. 

2.3.4. 1 Data Breakdown 

Before scoring the compiled critical Incidents from SL-3 and SL04, the three 

principal personnel assigned to Critical Incidents Analysis reviewed the total body 

of data gathered. It was apparent that the incidents reported in the transcripts fell 
generally into two large categories: 1) reports of errors on the part of the flight 

crew or the ground, and 2) comments (not associated with an error) criticizing or 

complementing various procedures, schedules, operations, or hardware. 

As the remarks were reviewed, a more detailed picture of the two categories 

emerged and was refined into the following sets of categories and subcategories: 




ERRORS 


Errors were divided into four types: 

• Commission error - Flight crew performed some operation that was 
not supposed to be performed, or performed an operation inacurrately . 

• Omission error - Flight crew failed to perform a directed operation. 

• Inversion error - Flight crew performed two or more operations out 
of sequence from the way they were scheduled. 

• Time displacement error - Flight crew performed some operation 
early or late, or ran out of time and could not complete an operation. 

These were then subclassified, wherever the data indicated, as pertaining 

to either hardware or procedures, as shown below. 

Hardware 


• Controls, by type 

• Displays, by type 

Procedures 


• Scheduling or instructions — as trained, contained on the cue 
cards, pad, provided by way of special voiced instructions, etc. 

• Operations - performing or failing to perform an operation pertain- 
ing to the ATM panel . 

• Time displacement ~ not performing an operation at the scheduled 
time. 

COMMENTS 


These data were divided into the same hardware and procedural group- 
ings used for subclassifying error categories, e.g.. 

Hardware 


• Controls 

• Displays 
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Procedures 

• Scheduling/Instructions 

• Operations 

• Time Displacement. 

The difference between "comments*' and the incidents scored as errors was 
that the critical remarks scored as "comments" were not Immediately and obviously 
associated with an error. They were mostly remarks about the hardware, schedules, 
or procedures, i.e., comments about inadequacies in specific hardware parts, sugges 
tions for improving Instructions, beneficial discoveries about hardware auxiliary 
capabilities, questions about proper procedures, and requests for clearer instruc- 
tions. 

2. 3, 4. 2 Basic Scoring Procedure 

After review of all the Critical Incidents Raw Data Score Sheets, the three 
analysts principally Involved in reading the transcripts conferred to score each 
reported incident into the appropriate categories and subcategories described in 
Paragraph 2.3.4. 1, Remarks were scored as, 

• Error (+ type, i.e., commission (C), omission (O), Inversion (I), time 
displacement (TD)) 

• "Comment", a remark not associated with an error (recorded as a "positive 
comment" (C+) or "negative comment" (C-)). 

It must be realized that every scored critical incident evolved out of some remark 
made by either the flight crew or the ground support team, but the term "comment" 
in the context of critical incidents analysis scoring has a special meaning, as indi- 
cated by the discussion of the term in Paragraph 2 . 3, 4. 1 . 
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The procedure for identifying, classifying and scoring critical incidents is 
shown in Figure 2-10. 

Some examples of incidents and scores are provided below to show more pre- 
cisely how scoring criteria were applied. 

Incident Scored as : 

Example #1: Crewman began S052 Commission Error, 

Day 231-16-01-02 Building Block before Time discrepancy 

ESR by mistake. (C, TD) . 

Example #2: Ground informed crew Omission Error, 

Day 232-1 0-M5-05 that the Voltage Detector Operations 

had been left in ENABLE (O, O) 

overnight. 

Example #3: Crew remarked that Negative Comment, 

Day 226-28-06-20 S055 Detector 5 kept HV Detectors 

kicking off during MARs, (C-, Detectors) 

so they turned it off. 

Basically, these scores were entered Into master tables for SL-3 and SL-4 across 
from the experiment they pertained to and under the hardware or procedure type 
identified in the remark. Errors and "comments" were tallied separately, so discus- 
sion could take place with regard to; 1) noted (or Inferred) error frequency, and 
2) non-error associated remark frequency. Total Incidents (Including both noted 
errors and non-error remarks) were also computed and studied. 

In addition to the data scored on the master score sheets for SL-3 and SL-4, 
a running log was maintained for the purpose of recording single event critical 
incidents. That Is, all voice transcript comments that were, by themselves, con- 
sidered to be important were noted and saved for future discussion. 
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*Voice transcripts revealed what time the remark was made and, therefore, the 

analysts had only an approximate Indication of what time the Incident actually 
occurred. 


Figure 2 10: Flow Diagram of Critical Incidents Scoring Procedure (Shows major 

classifications and subclassifications of critical Incidents used in data breakdown 
and scoring . ) 
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The master tables for SL-3 and SL-4 Critical Incidents Analysis and results 
of the analysis are presented in Section 3.2. 
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DATA ANALYSES, RESULTS AND CONCLUSIONS 

This section addresses the analyses, results and conclusions of separate 
studies of two bodies of data. Fundamental to many parts of the discussions 
presented here are detailed descriptions of the analysis procedures and the ways 
they were used to extract the results and conclusions reported. Therefore, 

Section 3.1 contains discussions of procedures used, results obtained and con- 
clusions drawn during the descriptive statistical analysis and during the formal 
testing of hypotheses relating to the telemetry data. Section 3.2 contains dis- 
cussions of procedures, results and conclusions associated with the analysis of 
the voice transcript data. 

3,1 TELEMETRY DATA 

Before the raw telemetry data could be subjected to statistical analysis. It 
was necessary to tally the rows and columns of all of the individual data point 
score sheets, an example of which Is shown in Figure 3-1. This was done using 
time as an index, the data were pooled within each one-third of each of the two 
Skylab missions studied. Thus, three Data Summary Sheets (see Appendix B, 
Figure B-la - B-1F) were prepared for each mission. The summary totals for each one 
third mission period were, in turn, added with the other two one-third summaries 
for the same mission to provide a Whole-Mission Data Summary Sheet (Appendix B, 
Figure B-2a - B-2b) . Then, all four data summary sheets for each mission were used 
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to provide the error and error opportunity data necessary to calculate (by dividing 
error opportunities into errors, ) the error rates that appear in each cel! 

of the Error Rate Comparison Table (Table 3-1). As this table was the point of 
departure for all of the statistical analyses, and as it is extensively referenced 
throughout this report, a detailed description of it is presented here. 

Upon examination, it is apparent that the comparison table (Table 3~1 above) 
is made up of nine columns of 38 rows (342 individual cells) . Each row has been 
assigned a number from 1 to 38 and each column has been given a letter from 
A to I . Arrangement of the rows and columns within the table is intended to 
facilitate visual as well as mathematical analysis. For example, columns A, B 
and C contain the error rate data for all the measurement parameters from the 
first, second and third ono-thirds of SL-3, respectively. In the same manner, 
columns D, E and F contain the error rate data for SL-4. Columns G and H 
contain the overall error rates for SL-3 and SL-4, respectively. It is, therefore, 
a relatively easy matter to compare the data from the first mission across time 
with data from the second mission. It Is also easy to compare error rate levels 
for each parameter from mission to mission by simply referring to columns G 
and H . 

Each row contains the error rate data relevant to the measurement para- 
meter indicated at its extreme right hand side. Of the 38 rows, 37 are used to 
describe the same population of data in four distinctly different ways: 

1) Rows 1 - 16 — Controls and Displays, separately 

2) Rows 17 - 24 -- Controls and Displays, combined 
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3) Rows 25 - 35 — Hardware Type 

4) Rows 36 - 37 — Sequential vs Single Operations 

Specifically, rows 1 through 8 include error rate data on the controls for each 
of the ATM experiments. Rows 9 through 16 contain complementary display data 
for the same experiments. Rows 17 through 24 contain error rates calculated 
from exactly that same body of data, but grouped into control and display 
errors combined for each ATM experiment. Rows 25 through 35 contain a third 
set of error rates calculated for the original data population. These error rates 
are for each specific kind of ATM panel control or display component Investigated. 
Rows 25 through 33 contain error rates for controls and rows 34 and 35 contain 
those for displays. One last error rate set describing the data population is grouped 
in, rows 36 and 37. Here, all ATM C&D activity Is transposed into error rates accord" 
ing to its membership in one of two classes: single or sequential operations. 

Row 38 was added to the table for com p«i risen purposes only. It contains 
'•normal ized" procedural deviation data that are unrelated to the ATM control and 
display hardware. Its method of computation and functional application is fully 
described later in the text. 

3.1.1 Descriptive Statistics 

A variety of related descriptive analysis techniques were employed in an 
effort to describe between — and within -- Skylab ATM experiment performance 
relationships. These were; 1) a direct, item by item comparison of Table 3-1 
cell data, 2) a search for ordering effects, and 3} an identification of all Table 3-1 
cells with error rate values of .05 or greater. 
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3. 1.1.1 Error Rate Cel! Comparison 

Overall Mission Performance - Comparison of overall mission performance on 
SL-3 and SL-4 (cells 17G and 17H of Table 3-1 ) , reveals that the SL-4 error rate was 
approximately half (.53) that of SL-3. Both the controls and the displays contributed 
to this difference (refer to cells IG, IH, 9G and 9H of Table 3-1 ) . While dramatic, 
this difference is nevertheless, in the direction expected, a fact attributable to the 
learning that took place on the part of the SL-4 crew and ground support personnel 
as a result of the previous mission experiences, longer available training time and 
improved procedures directives. However, when the error rates for both missions 
are plotted by thirds of mission (Figure 3-1), some unexpected differences appear 
in the profiles. The SL-3 error rates diminished as the mission progressed. This 
improvement in performance was expected, and could be explained as being due to 
the effect of learning and environmental adaptation, but, if learning and adaptation 
phenomena are used to explain the error rate in SL-3, how can the Increases that 
occurred across SL-4 be explained? For the answer to that question, it is necessary 
to return to the data. First, referring to Table 3-1 , it appears that cells 18E, 18F, 

21 E and 22F account for the increased error rates for the second and third one-thirds 
of SL-4. Checking further reveals that the high error rates registered in cells 18E 
and 18F were due entirely to high Hal experiment control error rates (cells 2E and 
2F in Table 3-1) . Therefore, an in-depth search of the raw data score sheets was 
made to identify the reasons for the high cell 2E and 2F error rates. (Note: cells 
21 E and 22F were excluded from investigation at this point because of their relatively 
lower error rates and their disjointed relationship to each other.) As a result of the 
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Figure 3-2: SL-3 and SL-4 Mission Errori 



Figure 3-3: SL-3 and SL-4 Mission Error 
Rate Profiles (Revised) 


Error 

Rate 



Mission Time Period 

Figure 3''4; SL-3 and SL-4 Display Error 
Rate Profile 
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in-depth data search. It was discovered that, during both the second and third one- 
thirds of SL-4, there were uninterrupted periods of time when the crew's selection 
of an Hal frames per minute rate was always In error. For the second one-third of 
that mission, the error period spanned data points 253 to 256 and accounted for all 
four (4) Hal errors registered. For the third one-third of the mission, the period 
covered data points 293 through 309 and accounted for all thirteen (13) Hal errors. 

If the Hal errors registered during the two periods of time investigated had 
been due to some panel design deficiency, they would be expected to have been 
scattered more or less randomly through the data. That they were not is indicative 
of some study data source deficiency -- that is to say. It is likely that the SL-4 crew 
was either instructed to take Hal pictures at a rate other than the one indicated in 
the Building Blocks or that they decided on their own to take pictures at a different 
rate. In either case, there is little justification for scoring these events as errors 
during those two periods. Thus, the error rates for Table 3-1 cells lE, IF, IH, 

2E, 2F, 17E, 17F, 17H, 18E and 1 8F were recalculated. All of the Hal error rate 
cells (Table 3-1 cells 2E, 2F, 18E, 18F) changed to a value of zero. More importantly, 

I 

cells 17E and 17F (Table 3-1), reflecting overall ATM experiment error rates for 
the second and third one-thirds of SL-4, were changed to .019 and .017, respectively. 
This implies that there were no practical differences in error rates between the first, 
second and third one-thirds of SL-4, so there appeared to be no learning or adapta- 
tion effects experienced by the SL-4 crew. This finding, together with the low over- 
all error rates experienced on SL-4 suggests that prelaunch training for SL-4 was 
more effective and more complete than was that for SL-3 (irrespective of differences 


3-8 



in levels of task difficulty and complexity) . Before such a statement could be made, 
however, it was necessary to check the SL-3 data to ensure that no hidden study 
discrepancies had accounted for the SL-4 error rate profile. When that check was 
made, it was found that the SL-3 errors were distributed with appropriate homo- 
geneity throughout the mission. Therefore, it appears likely that most of the errors 
scored for SL-3 were due more to the crew-ATM system interaction effects than to 
any study artifact. Thus, on the basis of the adjusted overall mission error rate 
profiles (Figure 3-2, above) , some combination of the following may apply: 

• SL-4 crew training was more effective and complete than that received by 
the crew of SL-3. 

• SL-3 tasks were more difficult than SL-4 tasks. 

I 

I 

• SL-4 crew environmental adaptation occurred prior to their commence- 
ment of ATM experiment operations. i 

• SL-4 ATM experiment work tasks increased in difficulty at a rate which 

offset any increase in crew proficiency. \ 

individual Display Performance - While the overall error rate profiles Of the 

'ft 

two crews differed, both crews were similar In that they greatly reduced their . 
across-time error rates while using displays (Figure 3-3, above) . In spite of this, 
neither crew was able to perform any better than 1/3 to 1/5 as well on displays than they 
did on controls (Table 3-1, cells 9C, 9H, IG and IH). In looking for explanations 
for those statistics, it was necessary to look directly at the display population charac- 
teristics. First, the number of display types contributing data to this study was 
small — only three. These included timing devices, experiment ready-operate 
lights, and the S055A grating position indicator. Second, these displays served 
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only two functions timing and positioning. 

Timing was accomplished both through the use of timers and by checking the 
status of ready-operate lights. Many of the operations that involved the use pf 
timers (i.e. , displays) were terminated automatically. These were part of a sequen?- 
tial operation, which, judging from the sequential operation error rate data (Table 3-1 
cells 3A through F), contributed little to the displays problem. Indeed, when the 
individual data points are examined, this is borne out. The remaining operations 
Involving the use of timing devices and ready-operate lights were manually initiated 
single operations. A check of the single operation error rate cells (Table 3-1, 
cells 37A through F) indicates this is where the problem lies. That manually timed 
operations, were often poorly performed is not surprising, since the displays used 
on this panel were designed to be of low target value. 

Incorrect setting of the S055A grating position involved a totally different 
problem. That operation was always performed sequentially, and, furthermore, 
the display was generally closely monitored to ensure its accurate setting. Never- 
theless, many grating setting errors occurred. After checking the individual 
data point score sheets, no doubt remains that some few of the positioning errors 
resulted from incorrectly interpreting directive information. Most of the errors 
of this kind, however, were due to the dynamics of the display itself. These 
occurred in the following way: The directive information specified a grating setting 
on a particular line. Yet, when the S055A mode switch was set In REFERENCE and 
the start/stop switch was activated, the last digit on the grating position indicator 
changed so rapidly that its value was impossible to read. Thus, the operator was 
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forced to use the tens digit together with an anticipatory response in order to approx 
mate the correct setting. Anticipatory response inaccuracies often presented the 
operator with two distasteful choices. First, if he undershot the mark, he was faced 
with placing the mode select switch In GRATING, SINGLE and advancing the display 
number one count at a time with the start/stop switch. Because of this cumbersome 
procedure. It is likely that the crew tried, instead, to go as close to the mark as 
possible while in the REFERENCE position. Unfortunately , trying for the close shot 
often led to an overshoot predicament. In that instance, the operator could decide 
he was close enough or elect to go back through the whole grating select procedure 
(incurring a 5 min. delay) . Thus, these overshoots, coupled with operator reluc- 
tance to repeat a cumbersome and time consuming procedure to arrive at a correct 
grating setting, led to most of the grating position errors. 

Overall Display Performance - Having addressed the finding of overly poor 
display performance, it is now appropriate to return to the discussion of display 
error rate profiles. As previously stated, display error rates do decrease across 
time in both missions. In fact, the decrease in both cases is sufficiently consistent 
and dramatic to raise some question about the extent to which ground training (i .e, , 
their previous learning as applied to crew ability to use the displays provided) 
prepared them for inflight display operations. However, the general inability of 
these displays to serve their intended function is the overriding factor. Since that 
has already been discussed, the Issue will not be pursued further here. 

Additional Hardware Problem Areas - In addition to the problem areas already 
discussed, there are three other points associated with the Error Rate Comparison 


3-11 



IIW^mHinx 


Table (Table 3-1) that must be covered. One is the obvious discrepancy between 
the SL-3 and SL-4 illuminated status indicators error rates (Table 3-1, cells 35G 
and 35H) . Such a discrepancy is sufficient cause to look for some contributing 
factor. In this case, the factor seems to be the data themselves. Specifically, it 
is the number of times that an illuminated status indicator was used during the SL-4 
Mission, e.g., a use frequency of 13 simply does not justify placing great confidence 
in the error rate calculated. 

The second point to be covered relates to six-position rotary switches. Those 
switches were also associated with a dramatic shift in error rates from SL-3 to SL-4 
(Table 3-1, cells 31G and 31H) . Unlike the case of illuminated status indicators, 
however, these controls were employed a sufficient number of times during both 
missions to justify some confidence in the error rates calculated. Unfortunately, a 
close second look at the individual data point score sheets did not uncover any reason 
for this shift, so it must be written off as an error shift attributable to differences 
in crew training effectiveness, deliberate operator deviation from directed procedure, 
unrecorded changes of instructions, or isolated accidental errors. 

Procedural Deviations - Procedural deviations is the third point associated 
with the Error Rate Comparison Table that must be discussed. While Included In 
the table, in truth, these data are more "normalized" than they are error rate quanti- 
ties. That is, while the error rate data represent the quotient of errors over error 
opportunities, procedural deviation rates were calculated by dividing the total 
number of procedural deviations by the total number of control plus display error 

opportunities ( . Procedural deviations could not be classified as 

Total C£D EO 
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errors, nor could they enter into determination of the number of error opportunities. 
Therefore, at first glance, it would not seem appropriate to divide one by the other. 
Yet error opportunities are the best available measure of ATM experiment work 
density, so dividing error opportunities into the number of procedural deviations 
should normalize the data and provide some insight into a procedural deviation 
"rate". 

Now that the procedural deviation number and its method of calculation have 
been explained, some effort will be made to define the term "procedural deviation". 

A procedural deviation is any ATM experiment C&D activity involving two or more 
operations that is performed in place of, or in addition to, the activity required by 
official directive. In other words, a procedural deviation is activity not called for 
in the study reference documents, but which was almost certainly done on purpose. 
Recognizing this, it should be clear that there are at least three reasons for the 
occurrence of procedural deviations; 

1) The operator deliberately added or substituted the activity. 

2) Some unidentified source directed the operator. 

3} The operator misunderstood his operating instructions. 

If the first or second reasons are the predominant causes for procedural 
deviations, it seems plausable to expect the deviation density value to vary somewhat 
independently of the overall controls plus displays error rate for the same operating 
period. Conversely, if the procedural deviations occur primarily as a result of 
misunderstood instructions (procedural errors) It seems plausable to assume that 
the deviation density number would correlate positively (at least loosely) with the 
overall controls plus displays error rales for the same operating period. With this 
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rationale having been explained, it is now appropriate to turn to Table 3-1 to 
examine the actual procedural deviation density rates. When comparing the rates 
In cells 38A through C with those in cells 17A through C, a loose similarity of 
pattern becomes apparent. Thus, to the degree that the assumptions just advanced 
are sound, the procedural deviations recorded during SL-3 are due to misunder- 
stood or misinterpreted instructions. In other words, they are procedural errors. 
When comparing the procedural deviation density rates with the controls and displays 
error rates for SL-4 (see cells 38D through F and cells 17D through F in Table 3-1), 
a slightly different pattern emerges. At the outset and through the second one-third 
of SL-4, both rates remain more or less constant. Then, in the third part of the 
mission, the procedural density rate soars upward while the controls and displays 
error rate number remains at its former level. Again, to the degree that the assump- 
tions germane to this analysis are correct, the procedural deviations that occurred 
In the first two thirds of SL-4 are procedural errors. Those deviations occurring 
in the last third of SL-4, however, are deliberate procedural deviations. Thus, 
for all SL-3 and SL-4 time periods, save the last one-third of SL-4, there is statis- 
tical evidence that procedural errors did occur. Such evidence, in turn, suggests 
that there were some incidents of procedural directive functional breakdown. 


3. 1.1. 2 Ordering Effects 

After addressing the Error Rate Comparison Table (Table 3-1) cell-to-cell 

relationships, the table data (revised) were given a second look to check for mission- 

to mission ordering effects.* After rank ordering all the parameter error rates, only 
*Revised figures are noted in brackets I. 1 in Table 3-1 . 
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one such effect could be found, and that one had to do with toggle switches. Even In 
that case, only two members (3-position, F-F-F, and 3-position, M-SL-M) were in- 
volved, Considering the components and the kinds of errors involved, it is likely 
that the ordering effect observed was due as much to component function as to any 
particular physical component attributes. 

.3. 1.1. 3 Unacceptable Error Rates 

For the analytical purposes of this study, any error rate equal to or greater 
than .05 was considered unacceptable. Once Identified, these error rates (the 
circled quantities In Table 3-1) were used in a subtractive process to "drive out" 
their causative factors. Hal serves as a straightforward example of the process. 

Row 18 of Table 3"1 shows that 8 of the 9 cells contain unacceptable error 

rates. A quick look at rows 2 and 10 revealed (by the relative absense or presence 

of unacceptable error rates) that the problem is control related. Therefore, the 

Hal controls row of each Data Point Score Sheet (Figure 3-1, above) was consulted. 

In so doing, it was determined that all of the Hal errors identified occurred in 

selecting the frames per minute picture rate, an operation that involves the setting 

of a three-position, F-F-F, toggle switch. This knowledge enabled a factoring 

out of both the error and the error opportunity numbers from the appropriate 

half-cells under the H«1 controls (alone), under the Hal controls and displays 

(combined) and under the three-position toggle switch (F-F-F) headings. Then, the 

error rate for each cell impacted was recalculated. If the recalculated rate fell 

below .05, then the factor removed was credited as the one responsible for the 

unacceptable error rate. During the course of this analysis, if there had been an 
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instance where the removal of no single factor would drive the error rate below 
.05, then as many contributing factors as possible would have been identified. That 
done, the fewest number of those factors that, extracted together, would have dropped 
the error rate to below ,05 would have been credited with the unacceptable error 
rate. Fortunately, in this study, every unacceptable error rate cell could be 
accounted for by removing just one factor. Those factors together with a compendium 
of the Table 3-1 cells whose error rates they drove above .05 and presented in 
Table 3-2. 

Table 3-2: Factor Classification for 
Error Rates (Cell Values) Exceeding 5% 


1 . TIME DISPLACEMENT FACTOR 

5B, 5G, 6A, 6C, 6F, 6G, 8B, 9B, 9C, 9E, 12A, 12G, 13E, 14F, 14H, 141, 15B, 
15C, 15E, 21B, 21E, 22A, 22C, 22F, 22G, 24B, 34B, 34E, 35A, 35B, 35C, 35G, 
351, 37A, 37B, 37C, 37D, 37E, 37G, 37H and 371. H = 41 

2. S055A GRATING POSITION FACTOR 

9A, 9D, 9C, 9H, 91, 15A, 15D, 15G, 15H, 151, 23A, 34A, 34D, 34G, 34H and 
341. N = 16 

3. Hal FRAMES PER MIN. FACTOR (Revised Figures) 

IB. 2A, 2B, 2C, 2C, 21, 17A, 17B, 18A, 18B, 18C, 18G, 181, 27B, 27F, 27G, and 
36B, N = 17 

4. MODE SELECT FACTOR 


0 

5-position rotary sw: 30C, 30D, 30F and 30G 

N = 4 

o 

6-position rotary sw: 31A, 31B, 31G and 311 

N = 4 

o 

3-posltion toggle sw (F-F-F): 5A 

N = 1 

o 

4-positlon rotary sw: 29F 

N = 1 

o 

8-position rotary sw: 32D 

N = 1 

o 

11-posltion rotary sw: 33E 

N = 1 


TOTAL 

N = 12 
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Table 3-2: Factor Classification for 

Error Rates (Cell Values) Exceeding 5% 
(Continued) 


5. DATA POINT #6 FACTOR 

27A N - 1 

The error rates above .05 in 85 of the Table 3-1 cells can be attributed to 
the presence of the five factors ranked by cell impact in Table 3-2. Indeed, the 
unacceptable error rates In 84 of those cells can be attributed to the first four factors 
The fifth factor accounts for the unacceptable rate in the remaining cell. But, being 
the performance on an isolated task in the early period of SL-3, that factor is not 
representative of any recurring problem and will not receive further discussion. 

3 . 1 . 1 . 3 . 1 Time Displacement Errors 

Time displacement errors by far accounted for the largest number of error 
rate cel Is 41 in all, or 47.1% of the 87 total . In addressing this problem, at 
least four contributors (excluding hardware malfunctions and isolated accidental 
errors) should be recognized. They are; 

1) Deliberate deviations 

2) Directive functional breakdown 

3) Procedural problems 

4) Display functional breakdown 

1) Deliberate deviations - In many complex tasks where the operator has 
decision making authority, deliberate deviations from established procedure are 
likely to occur. While these deviations are not real errors, when they involved 
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only one operation, they were scored as errors during this analysis. Therefore, 
they certainly had some Impact on the overall error picture. Unfortunately, the 
telemetry data do not provide sufficient resolution to allow assessment of problem 
magnitude. 

2) Directive functional breakdown - This category includes such error 
sources as misread checklists or misunderstood voice communications. And, while 
no specific examples have been extracted from the telemetry data analysis to support 
the argument that this category contributes to time displacement errors, for both 
SL-3 and SL-4 there are telemetry data that establish its role in related kinds of 
errors (e.g., inversion of order, numerical digit transposition and Hal frames/min) . 
Add to this prima facie evidence the cluttered Building Block format and some voice 
transcript comments, and indeed, there is a good face validity argument for Infor- 
mation breakdown's contribution to time displacement errors. 

One of the likely problems in this area is the way operational directives are 
presented. In the Building Blocks, for example, manual termination is signified 
by the use of a dashed line in place of the normal solid one. Because of the physical 
characteristics of both lines, the distinction can be easily missed by a "pushed" 
crewman. Another Building Block problem is the method used to identify activity 
initiation times (i.e., by checking the alignment of noncontinuous lines with hash 
marks across the top and bottom of a page) . Still another source of Building Block 
information breakdown that contributes to time displacement errors is the lack of 
prominence given to experiment run time. 
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The impact of voice communications and other directive sources on time 
displacement errors could not be investigated using the telemetry data base. Insights 
pertaining to this area must be derived from the Critical Incidents Analysis, Section 
3.2. 

3) Procedural problems - Procedural problems that lead to deviations from 
expected timelines can be caused by factors such as inappropriate mission scheduling, 
extreme work density. Incompatible work task arrangement and equipment malfunc- 
tions. in fact, the telemetry data together with the voice transcripts provided 
evidence that all of those factors played some role in contributing to the time dis- 
placement problem. The telemetry data alone, however, are of little utility In deter- 
mining the exact role of each factor . Therefore, discussion of these factors must 
also be seated in the Critical Incidents Analysis. 

4) Display functional breakdown - This is one area on which the telemetry 
data provide an abundance of information. For example, the data show repeated 
instances where S055A was not manually terminated in conjunction with the auto- 
matic termination of other experiments as required. Yet when this action is required, 
the related Building Block directives are among the clearest of all directives presented 
Thus, when manual experiment termination is keyed to the automatic termination of 
another experiment, it appears that the blame for any time displacement errors can 

be laid at the feet of the displays involved. For an in-depth description of the 
particular problems associated with the use of those displays, refer to Section 3. 1 . 1 . 1 . 
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3. 1.1. 3. 2 Setting the S055A Grating Position Incorrectly 

The incorrect setting of the S055A grating position placed second in impact 
behind time displacement errors. At least three factors contributed to the incorrect 
setting of that grating. They were; deliberate deviations, directive functional 
breakdown, and display to user functional breakdown. Deliberate deviations pose 
no problems and will not be discussed further. Directive functional breakdowns, on 
the other hand, were a problem. In at least one instance where the Building Block 
operating procedures called for the selection of six different grating positions in 
order, the order of actual performance was incorrect, and one of the ordered grating 
positions was omitted. 

Display functions breakdowns for the S055A grating position indicator have 
already been fully treated in Section 3.1.1 and will not be treated further. 

3. 1.1. 3. 3 Hal Frames Per Minute Errors 

Errors in selecting the proper number of frames per minute for Hal was the 
third greatest contributor to the number of cells containing error rate figures above 
.05. Because the .000 error rate (revised) incurred while operating this frame rate 
control during SL-4, it is not likely that the control characteristics themselves were 
responsible for the high SL-3 rates. It is more likely that some combination of 
deliberate deviations and directive functional breakdown produced the rates above 
the .05 level. Here again, deliberate deviation are of no concern. Directive to user 
information transmission breakdowns, on the other hand, are of concern. And, for 
Hal, any Information transmisssion breakdown most assuredly was the result of 
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the position relegated to frame rate information In the Building Block format. 

3. 1.1. 3. 4 Mode Selection Errors 

The remaining cells containing error rate above .05 can be accounted for by 
a mode select control factor. Most probably that factor also contains such contributors 
as intentional deviations and directive functional breakdown. However, it seems 
apparent from looking at the hardware variations making up the populations of mode 
select controls (Table 3-2, above) that control function seemed to play a greater role 
in error rate than did control type. 

3.1.2 Formal Statistical Analysis, Testing Eleven ATM C&D-Related Hypotheses 

Up to this point, information about the analysis of SL-3 and SL-4 telemetry 
data has focused on the signostic or analytic approach taken in studying the various 
components and characteristics of the control panel. Error data so obtained and the 
consequent effects of those errors on mission performance served as the criteria for 
focusing study to specific panel elements on operations. Following that portion of 
telemetry data analysis, and using data derived from it, a formal analysis which 
tested eleven hypotheses was conducted. This section (3.1.2) comprises a report 
of that formal analysis. 

3. 1.2.1 Hypothesis Development 

Based on experience in designing, testing, and using the panel, analysis of 
SL-2 data, and general human engineering design principles, URS/Matrix developed 
a list of 25 hypotheses of interest. The data requirements for testing each hypothesis 


3-21 


rm mHTrii 


were compared with the data telemetered during SL~3 and SL-4 relevant to the ATM 
controls and displays. Hypotheses were then categorized in terms of their being 
directly testable, indirectly testable, or not testable. The directly and indirectly 
testable hypotheses were discussed with NASA technical representatives. 

Table 3-3 contains the hypotheses determined to be of interest and testable, 

given the data available from SL-3 and SL-4. 

Table 3-3; ATM Assessment Hypotheses 


HARDWARE RELATED 

1 . The use of three-position toggle switches produces a significantly higher error 
rate than does the use of other types of toggle switches. 

2. The use of rotary switches with positions numbering more than four produces 
a significantly higher error rate than does the use of rotary switches with 
four positions. 

3. The use of rotary switches with positions numbering more than four produces 
a significantly higher error rate than does the use of other types of switches 
used on the ATM panel. 

LAYOUT AND FUNCTIONAL ARRANGEMENT 

4. Experiments operated on the left side of the C&D panel produce a significantly 
higher error rate than do experiments operated on the right side. 

5. Operating Hal produces a significantly higher error rate than does operating 
any other experiment on the left side of the panel . 

6. Isolated actuations produce a higher error rate than do sequential operations. 

7. There is no significant difference in performance between operating S082A 
and operating S082B. 

8. Operating experiments with high C&D layout similarity (e.g. , Ha2, S052, 
S082A, S082B) produces higher error rates than does operating experiments 
with dissimilar C&D arrangements (e.g., S056, S055A, S054) . 
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Table 3- 3: ATM Assessment Hypotheses 
(continued) 



9. Operating the lower half of the ATM Panel produces a significantly higher 
operational error rate than does operating the upper half of the panel. 

TIMELINE AND OPERATIONAL PROCEDURES HYPOTHESES 

10. SL“4 operations produce lower error rates than do SL~3 operations. 

11. The first 1/3 of each mission's operations produces higher error rates than do 
the remaining 2/3 of each mission. 

3- 1.2. 2 Hypothesis Testing and Results 

For each hypothesis, appropriate statistical tests were identified and run. To 
meet the constraints (e.g., repeated measures or correlated samples, non-normal 
distribution, missing data) imposed by the data, both chi-square and analysis of 
variance statistical tests were employed (Table 3-4) . Table 3-5 presents the number 
of errors vs error-free operations plus error rate for each hypothesis. 

3. 1.2. 2.1 Hardware Related Hypotheses 

None of the hypotheses dealing with controls were accepted. The variation in 
errors appeared to be less a function of the specific type of control device than of 
the procedural and functional envelopes surrounding the control. Certain switches, 
e.g., five-position rotary, three-position toggle (F-F-F) have relatively high error 
rates (.048 for the former, and .046 for the latter), but these rates were not reflected 
consistently in other similar switches. This, combined with the observation that 
high errors for any given control did not occur consistently across time periods for 
either mission, reinforces two principles of C8D design. 
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1) The adequacy of a given switch in a complex control panel is only partially 
dependent on the design of the switch. The relation of the switch to other 
control elements, layout, sequence, functional procedures, workload, or 
the general operating envelope around the switch appears to be equally 
important. 

2) Evaluation of control hardware to be used in complex panels or operations 
must be performed within the full operating envelope, where the inter- 
actions noted above can occur. 

While not a formal hypothesis, a question of considerable interest is the 
performance of operations involving the use of controls versus those involving the 
use of displays. The errors and error opportunities were summed across missions 
for controls and displays and tested. The resulting chi-square of 38,72 (1 df) is 
significant well beyond the .001 level. Display operations, with an error rate of 
. 10 relative to the .032 rate for controls operations, clearly contributed heavily to 
the ATM panel error rate. 

3. 1.2. 2. 2 Layout and Procedural Related Hvootheses 

Five experiments, H«, S056, S082A, S082B, and S052, were on the left 
side of the ATM Panel, two experiments, S055A and S054, were on the right. 

As expected, the experiments comprising the left side had significantly (.01) 
higher error rates than the experiments on the right side. 

As noted in an earlier section of this report and confirmed in Hypothesis =5, 
Hot contributes heavily to this result. Fifty-six percent of the error for the five 
experiments on the left side of the panel occur while operating Hc». When Ha 
data are removed, the left side error rate Is .0279, and the right side is .020. 
Chi-square shows no significance for this difference. 
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Thus, left-right location appears to be a relatively minor contributor to 
error rate. The significant difference is attributable to one deviant experiment 
rather than left-right location per se. 

For Hypothesis #5, the Hot error rate (.1295) was compared with the error 
rate for all other experiments comb^ined (.0296). The chi-square of 80.51. was 


highly significant (p< .001} and clearly demonstrated the contribution of this 
experiment to mission error. Thi/s finding must be kept in mind when looking 
at all other testing using data compiled by experiments, since Hoi data tends to skew 
such distributions . / 

Sequential operations exhibited a significantly (p < .001) lower error rate 
than isolated or single operatiorys. The design implication of this result is that 
establishing a chain of control activities even if they are only marginally related, 
should help minimize error. Accomplishing this necessitates extensive attention 
to the procedural/instructlobal aspects of the control panel system design. 

Hypothesis #7 was ^cepted; there were no differences in error between S082A 
and B. These two exper/iments had the highest degree of similarity, in terms of 
location, layout, and procedures, and this was expected to contribute to errors of 
interpolation and l^ation displacement. The errors which did occur did not bear 
out this expectati/on considering only S082A and B. 

When all/ experiments were categorized on a similarity/dissimilarity dimension 
(Hypothesis/#8) , a significant error difference appeared. This finding must be 
viewed cayutiously because of judgmental assignment to the category and because of 


the skewing affects of Hal already discussed. The experiments included in high 


3-25 


irrpi mHTni 


similarity were: Hal, S052, S082A, and S082B, and In low similarity: S056, S055A 
and S054. 

In spite of this, the result provides support based on operational data that 
conspicuity and discernabilily must be considered in the design of controls and 
displays. As other hypotheses have implied, so this one also points to the import- 
ance of considering individual C&D and functional sub-units of a panel within their 
total operating envelope in design. 

The final hypothesis in this group tested the exeriments in the upper half 
of the panel, Hal, S056, S082A and S055A, and lower half of the panel, S082B, S052 
and S054. There was not a significant difference. 

3. 1.2. 2. 3 Timeline and Operational Procedures Hypotheses 

The differences between SL-3 and SL-4 were tested in several ways. Hypoth- 
esis #10 was tested using which showed the SL-3 error rate significantly (p< .001) 
greater than the SL-4 error rate. Hypothesis #11 was not accepted, the x^ between 
the first 1/3 and remaining 2/3 of each mission did not provide an adequate test of 
the data . 

Therefore, a 2x2 analysis of variance for repeated measures (Ref. Appendix A, 
Weiner, 1962) was used to test missions and time periods. The test was run using 
both transformed (x + .5, Ref. Appendix A, Edwards, 1962) and raw data. Mean error 
rates were calculated for each control and display category. This was done to alleviate 
the skewing effect of H« when the data were summarized by experiment. Table 3-6 
gives the results of the test. 
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Table 3-4: Summary of Results for Hardware, Layout, 

and Timeline Hypotheses for SL-3 and SL~4 


Hypothesis 

Statistic 

Result 


P Less 

Than 

1 . 

Chi-square 
(X 2 ) 

1.573 

1 

.22 

N.S, 

2. 

X ^ 

2.01 

1 

.15 

N.S. 

3.- 


.60 

1 

.80 

N.S. 

4. 

x2 

8. 95 

1 

.01 

* 

5.- — -— 

x2 

80.51 

1 

.001 

•k 

6. 

x^ 

33.57 

1 

.001 

* 

7— 

x2 

.687 

1 

.45 

N.S. 

8. 

x^ 

20.47 

1 

.001 

* 

9. 

x2 

1.22 

1 

.28 

N.S. 

10. 

F (Anova) 

5.00 

1 

.05 

* 




and 






20 




x2 

13.67 

1 

.001 

■ft 

11 






SL-3 

X ^ 

,66 

1 

.80 

N.S. 


Sandler's A 

1.18 

6 


N.S. 

SL-4 

X ^ 

2.81 

1 

.09 

N.S. 


Sandler's A 

.488 

6 


N.S. 
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Table 3-5: Data 

for Each Hypothesi 

is 





Error-Free 

Error 

Significance 

Hypothesis # 

Parameters 

Errors 

Operations 

Rate 

of Difference 

1 . 

3-position toggle 

90 

2208 

.0392 

Not 


All other toggle 

0 

62 

.000 

Significant (N .S . ) 

2. 

4-position rotary 

1 

126 

.007 

N.S, 


All other rotary 

24 

626 

.037 

3. 

5 or more position 

24 

602 

.0383 

N.S. 


rotary 






All other switch 

89 

2752 

.0313 

N.S. 


types 





4. 

Left side of panel 

77 

1474 

.049 

.01 


Right side 

73 

2321 

.030 


4. 

Left 

34 

1185 

.0279 

N.S. 

(With Ha 

Right 

73 

2321 

.030 

removed) 






5. 

Ha 

43 

289 

.1295 

.001 


Other experiments 

107 

3506 

.0296 


6 . 

Single operations 

25 

197 

.113 

.001 


Sequential operations 

125 

3598 

.033 


7. 

S082A 

3 

221 

,013 

N.S. 


S082B 

6 

196 

.029 


8. 

High similarity 

64 

976 

.0615 

.001 


Low similarity 

86 

2819 

.0296 


9. 

Upper half 

118 

2820 

.040 

N -S. 


Lower half 

32 

975 

.032 


10. 

SL-3 

100 

1933 

.0492 

.001 


SL-4 

50 

1862 

.0262 


11 . 

First 1/3 

44 

761 

.055 

N.S . 

(SL-3) 

Last 2/3 

56 

1172 

.048 


11. 

First 1/3 

13 

721 

. 018 

N.S. 

(SL-4) 

Last 2/3 

37 

1141 

.031 
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Table 3-6: 

Summary of Analysi 

is of Variance 


Source of Variation 

SS 


MS 

F 

Between 

.05 

21 



Missions (SL-3/SL-4) 

.01 

1 

.01 

5.00* 

Within 

.04 

20 

.002 


Within 

.06 

44 



Time Periods 

.01 

2 

.005 

4.54* 

Time by Mission 

.001 

2 

.0005 

.45 

Time within Mission 

.0499 

42 

.0011 



*p < .05 

The significant difference between errors in SL-3 and SL-4 was confirmed. There 
was also a significant difference between time periods. This indicated that the 
differences were not In the periods predicted in Hypothesis #10. Also, there was not 
a significant interaction between time and missions. When the error rates are plotted 
as in Figure 3-2, above, the lines converge, but this Is not statistically significaht. 

To determine in greater detail where the significant differences occurred, 
chi-square was used to compare SL-3 and SL-4 for each time period. The results 
were: 


Time Period 1 p .001 

Time Period 2 p< ,06 

Time Period 3 p .80 

Within each mission, time periods could only be compared using a statistic 

for correlated samples. Recently, Sandler’s A has been shown mathematically 

equivalent to the test for correlated samples (Ref, Appendix A, Runyon and Haber, 

1970), so this was used. Table 3~7 shows the results. 
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Table 3^7: Differences Between Time Periods in SL-3 and SL-4 


Time 

Period 

SL-3 

SL-4 

1 vs 

2 

N.S. 

N.S. 

2 vs 

3 

p < .001 

N.S. 

1 vs 

3 

N.S. 

N.S. 


The contribution of Hci accounts for the difference in time periods. When 
Sandler's A was recalculated without Ha data, the significant differences vanished 
This was verified by testing time for each mission using analysis of variance. 


In this case, each experiment was considered a block and the variance attributable 


to the blocks was not Included in the treatment variance. When the unusual variance 
contributed by Ha was statistically controlled, there was not significant difference 
between time periods for SL-3 or SL-4. Table 3-8 summarizes the analyses. 


Table 3-8: Summary Analysis of Variance for Time Periods 





SL-3 

MS X 

ss 

df 

SL-4 

MS 

±. 









Time Periods 

.0313 

2 

.0156 3.12 N.S. 

.00355 

2 

.001775 

1.24 N.S 

Blocks 

.0212 

6 

.0035 

.01836 

6 

.00306 


Residual 

.0605 

12 

.005 

.01719 

12 

.001432 


TOTAL 

.113 

20 


.0391 

20 




Turning to consider the experiments, only two showed a consistent pattern of 
errors over time for both missions. Table 3-9 gives the data for S055A, 
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Table 3-9; Error Rates for S055A on SL-3 and SL-4 by Time Periods 

Time -L ^ _L 

SL-3 ,073 .026 .021 

SL-^ .033 .014 .012 


Ha varied between time periods, but in the opposite direction from S055A, as 
shown in Table 3-10. 

Table 3-10: Error Rates for Ha on SL-3 and SL-4 
Over Time Periods 

Time 1 2 3 

.069 .257 .208 

.000 .125 ,167 


SL-3 

SL-4 


These were significant differences for several of the experiments from SL-3 to SL-4. 
Table 3-11 gives the results. Chi-square with x + .2 used. 


Table 3-11; Summary of Differences for Experiments 
Between SL-3 and SL-4 



Difference Between 
SL-3 and SL-4 


Ha 

S056 

S082A 

S082B 

S052 

S055A 

S054 


p < .05 

N.S. 

N.S. 

N.S. 

N.S. 

p < .02 

p < .05 


In all experiments, the error rate was higher for SL-3 than for SL-4 
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3. 1 . 2 . 3 Summary 

The formal analysis of the ATM C&D Panel depicts a situation in which there 
is a significant difference between the two missions. Within these missions, the 
major changes over time were attributable to one experiment, Ha , which was gener- 
ally characterized by an increasing error rate over time. Other experiments tended 
to decrease in error rate over time, but this was not significant. 

In terms of hardware design, the major significant finding was a difference 
between controls and displays. Differences in controls were not consistent enough 
within type of switch to show significance. This, In combination with the procedural 
and layout hypotheses (high vs low similarity, and single vs sequential operations) 
which were significant, highlight the importance of considering designing and 
evaluating components or functional sub units (i.e., an experiment) of a control/ 
display panel in terms of the entire operating envelope or circumstances. 

A primary use of this formal analysis appears to be as a means to focus or 
direct the evaluator, designer, engineer, or researcher, to areas, hardware or 
procedures, where a fine-grained examination can provide additional insight and 
design related information. Formal analysis, thus, is a useful and relevant but 
not all encompassing tool in diagnostically assessing man/machIne interface in 
operational settings. 
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3.2 CRITICAL INCIDENTS 

A detaiJcjd review of the SL-3 and SL-^ voice transcripts was performed . 
Mention or errors or other remarks were noted on Critical Incidents Raw Data Score 
Sheets, an example of which was provided in Figure 3-5. The example shows 
the type of information that could be noted or inferred from the transcripts (the 
limits of which have already been discussed in Sections 2.3.2 and 2.3.3) . 

3.2.1 Determination of Scored vs. Non-Scored Critical Incidents 

The Critical Incident Raw Data Score Sheets were being compiled concurrently 
with the scoring of the telemetry data, and, at the conclusion of voice transcript 
review, an effort was made to arrange the information so gained in a manner that 
would permit comparison of the results of the two analyses. The voice transcript 
data themselves fell rather readily into the two broad categories discussed in Section 

2.3 (i.e., ’’errors”, and "comments" not associated with errors], relating to the 
ATM panel. The exceptions formed a separate group of critical incidents from both 
missions which related more to hardware deficiencies or malfunctions not directly 
tied to the ATM controls and displays or the man/machine interface — the objects of 
primary scrutiny in this assessment study. This separate group was comprised of 
remarks relating problems encountered with the environment around the panel 
(e.g., low light levels making reading of instructions difficult), hardware short- 
comings or failures which affected but were not part of the actual ATM panel, and 
remarks exhibiting concern about resource depletion as it affected ATM operations 
(e.g. , low film supply in the experiments) . As assessment of the ATM C&D man/ 
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CRITICAL INCIDENTS RAW DATA SCORE SHEET 


MISSION TIME - \6 


JOP # 




BB # 


EXPERIMENT ^y0^j4 


5) NATURE OF COMMENT (POS OR NEC) 


6) BENEFIT OR SEVERITY (HIGH OR LOW) c/t^ W- 


HARDWARE OR.gROCEDURE^OF REFERENCE (SPECIFIC) 

J^YXyt-'b'-'Z cf 

Uc ^ 'i y\^. 

pyy^ if ^ 

OKIGINAL PAGE IS 
OF POOR QUALITY, 


Figure 3-5: Critical Incidents Raw Data Score Sheet 
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machine interface was of uppermost importance, these non-directly related remarks 
were left unscored on the master score sheets. Their repeated occurrence, however, 
in conversations about ATM operations between the crew and ground support required 
that they be dealt with in some way. While they were neither part of ATM C&D nor 
crew -control led, they appeared to be a logical contributor to some of the difficulties 
experienced by the crew during ATM operations. For example, if a filter became 
stuck in one position, due to presence of contamination, as occurred on S056 during 
SL-3, this had nothing to do with proper operation of the corresponding mode select 
switch or with the proficiency of the crew in operating the panel. It was, neverthe- 
less, a distraction during operations and also resulted in loss or degradation of 
experiment data. 


^'2*2 Reincorporation of Unscored Remarks 

Notations of such remarks were separated from the total collection of critical 
incidents during scoring, but they were studied to determine to what extent they 
may have impacted the man/machine interface. Those remarks representing parti- 
cularly troublesome or persistent problems were reincorporated into discussion of 
the experiments after the results derived from scored critical incidents (directly 
ATM-related) had been prepared for comparison with the results of the telemetry 
data analysis . 


3.2.3 Master Score Sheets for SL-3 and SL-4 Critical Incidents 

In all, 483 critical incidents were scored and entered into appropriate cells 
In Figure 3-6 for SL-3 and Figure 3-7 for SL-4. 
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3.2.3. 1 Scoring ATM Hard ware- Re la ted Incidents 


Examination of the horizontal axis of Figure 3-6 (SL-3) reveals the level of 
detail to which hardware incidents were classified, i.e., 

HARDWARE 

• 2-Position Toggle Switch (F-F) 


- Night Interlock 
^ Camera Power 

- Auto Sequence 


- Flare Auto 

- Grating Reference 

- BR-AL Det. 


3-Position Toggle Switch (M-F-M) 


- Start/Stop 

- Doors 

- Exposure 


- Wavelength 

- Mirror Position Camera 


3-Position (F-F-F) 


- Frames/Minute 

- Mode 

- Exposure 


- Crating 

- Detector 


3-Position (F-F-M) 


- Start/Stop 


Rotary Switch 


Picture Rate 
Mode 


- Exposure Range 

- Filter 


Counter 


- Grating 

- Intensity Data 


- Timer 

- Frames Remaining 


Experiment Related Equipment 


Monitor 


Pointing 
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CRITICAL INCIDENTS SCORE SHEET 

























































CRITICAL INCIDENTS SCORE SHEET 




HARDWARE ' PROCEDURES 





FOLDOUT FRAMi 
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The cells below these hardware types are divided into two parts by a diagonal line. 
Errors by classification code (C, O, I, TD} were entered into the upper part of the 
cells, while comments (C+ for favorable; C- for unfavorable) were entered in the 
lower part. The hardware-type categories are followed by a column separated into 
four parts, where the totals (by experiment number) for each of the four error 
classifications (i.e., Omission, Commission, Inversion, and Time Displacement) 
were entered. These totals refer only to total scored ATM hardware-related errors 
(by general error type) for a particular experiment. 

The "Errors" column is followed, in turn, by a column in which were recorded 
total hardware-related errors (of all four general classifications) In the upper half 
of the cell and total hardware- related comments in the lower half. 


3, 2,3.2 Scoring ATM Procedures-Related Incidents 

Following a separating space, a three-part column was used to record errors 
and comments by experiment number and by procedure category, i.e. , Schedule/ 
Instructions, Experiment Operation, and Operational Time Phasing, for Ha, S056, 
etc. Again, errors occupy the top of the divided cells, and comments occupy the 
bottom . 

To the right of the totals by procedure type is a final column in which totals 
for errors across procedure type and comments across procedure type for each 
experiment are entered Into the upper and lower cell halves, respectively. 

The vertical axis of the Master Score Sheet lists the ATM experiments followed 
by a slot for Experiment Pointing Control (EPC) and General Equipment. (The 
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General Equipment slot applies only to the Procedures column.) 

Last, a slot for totals of errors and of comments by hardware type and by 
procedures type for all vertically listed items is provided. 

3. 2. 3. 3 Totalling Incidents in Preparation for Analysis 

After the 483 scored critical incidents were entered into appropriate cells on 
the SL-3 and SL-4 master score sheets, they were totalled and analyzed for signifi- 
cant control/display problems and procedural difficulties. . During analysis, the 
data provided in the master score sheets was assessed by error total, comment 
total, hardware-related total (both broad and specific), procedures-related total, 
total errors and/or comments by experiment, and whole mission totals for SL-3 and 
SL-4. Emerging trends were charted and examined. Any cell with a value of 4 or 
more (i.e., 4 errors or 4 comments) were closely studied. Other cells, although 
scoring less than 4, were also studied if they were judged to represent remarks of 
substantial value to performance evaluation. 

The results and conclusions of that analysis are presented below by experiment 
number, along with mention, where appropriate, of Important non-scored incidents. 

To simplify organization, the results and conclusions derived from critical 
incidents analysis are divided by Skylab mission, SL-3 and SL-4. These two broad 
divisions are each subdivided into Hardware-Related Critical Incidents and Procedures 
Related Critical incidents. Discussion under both hardware and procedures is 
arranged basically by experiment number, with experiments remarked about most 
often being discussed first, because of the larger data base for those experiments. 
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by experiment number (and EPC) and were ranked by frequency of remark. Table 
3~12 provides a rank-ordered list of scored errors and comments followed by total 
scored critical incidents by experiment number for SL-3. Percentages and rankings 
mentioned in the ensuing experiment discussions are taken from those figures. 


3. 2.4.1 SL-3 Hardware-Related Critical Incidents 
S055A 

The S055A experiment was discussed via the audio link more frequently than 
any other SL-3 ATM experiment, and approximately 66% of the remarks scored on 
S055 occurred during the first nine days of the mission. 

With regard to scored errors, most of the errors that were recognized by the 
crew and ground personnel pertained to the high voltage (HV) detectors. Six out 
of the seven omission errors scored under the detectors were noted by the ground, 
and these all occurred on Day 228. A review of the scored comments relating to 
the detectors was made to try to determine a reason for this relatively large number 
of errors. It was discovered that 35% of the HV comments referenced detector #5, 
which malfunctioned early in the mission. These comments were encountered only 
in the first 10 mission days. 

Though not as frequently remarked on as the detectors, the Night Int^erlock 
and Crating Reference switches were also mentioned. All the errors connected with 
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Table 3-12: Rank-Order of Hardware Errors and Comments 
by Experiment for SL-3 Mission 


TOTAL 

TOTAL ERROR TOTAL COMMENTS CRITICAL INCIDENTS 


1) 

S055 

19 

1) 

S055 

19 

1) 

S055 

38 

2) 

Hal 

7 

2) 

Hal 

9 

2)- 

Hal 

16 

3) 

S056 

7 

3) 

S082A 

8 

3) 

S082A 

15 

4) 

S082A 

7 

4) 

S056 

4 

4) 

S056 

11 

5) 

EPC 

6 

5) 

S054 

4 

5) 

S082B 

9 

6) 

S082B 

6 

6) 

S082B 

3 

6) 

EPC 

9 

7) 

S052 

5 

7) 

S052 

3 

7) 

S052 

8 

8) 

S054 

4 

8) 

EPC 

3 

8) 

S054 

8 

9) 

Gen 

J. 

9) 

Gen 

1 

9) 

Gen 

3 


TOTAL 

63 



54 



117 
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these two components were discovered by ground support, (In fact, 41% of the 
errors reported on the experiment over the entire mission's length were reported 
by ground personnel.) The flight crew noted that one of the errors that occurred 
in connection with the Grating Reference switch was confusion between it and the 
S055 Start/Stop switch. 

Comments offered by the SL-3 crew also noted that the Raster/Scan counter 
was "acting up" and that the Intensity Counter had required malfunction procedures, 
but these comments did not include enough detail to conjecture their impact on experi- 
ment operation. 

H-Alpha 

The video monitors were the most frequently discussed hardware on the Ha 
experiment. Most scored incidents related to the quality of the video display images. 
During the first 7 days of SL-3 ATM operations, the crew commented on the problems 
they experienced in efficient Ha operation due to the rapid oscillations and "telescop- 
ing effect" they were getting on the video monitors. 

Critical incidents scored as commission errors under the Night Interlock switch 
were all noted by ground support. The voice transcripts contained several instances 
where ground personnel had to remind the crew to initiate this control. Some of the 
remarks referred to other times (apparently not recorded on the voice tapes) when 
the problem had occurred and attributed the errors largely to the crew's Ha instruc- 
tions. The ground reported that a change would be added to the cue cards to avoid 
future repetition of this commission error. 
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Of the hardware-related critical incidents mentioned by the SL-3 crew on 
S056, 45% were omission errors. The largest number of these errors pertained 
to the exposure switch. Although a change in position was indicated, this switch 
was left unchanged from previous procedures on several occasions, and the 
experiment was set up and Initiated with the switch incorrectly positioned. The 
crew remarked that they later realized the mode was to be changed but that they 
had forgotten to change the exposure setting. 

Although not scored on Figure 3-6, 24 critical incidents were recorded 
noting a mechanical failure of the filter wheel In the instrument. These failures, 
when noted, were usually accompanied by crew comments regarding restarting 
the experiment and missing data. The crew also noted that the problem distracted 
them from other operations, thus impacting not only 5056 data but also data gathered 
on other experiments which were run simultaneously with it. 


S082A 

The S082A instrument was remarked upon rather frequently. Experiment 
operations having to do with timed operations constituted most of the S082A hard- 
ware critical incidents, but with no consistent specific problem area. Approxi- 
mately one-third of the remarks (appearing in the first 7 mission days) tied in 
with S082A operation were associated not with actual S082A hardware but, 
instead, with 1) problems encountered with the Hci monitors, and 2} difficulty 
in using the monitors to identify solar features of interest. These comments, of 
course, did not represent errors, nor did they conclusively indicate S082A- 
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specific hardware deficiency. Rather, the comments seemed more to imply that 
the sun was calm and time was lost trying to find some "action" . 

S082B 

Almost half (44%) of the total hardware critical incidents scored on S082B 
concerned, again, the Hot monitors, as they applied to S082B operations. Diffi- 
culties identical to those noted on S082A were described. 

In addition, several comments were made by both the flight crew and 
ground expressing concern about the S082B film supply being very low. To 
conserve film, many scheduled auto sequences were omitted, with the unfortunate, 
concommitant loss of experiment data. 

S052 and SQ54 

Examination of the remaining ATM experiments, S052 and S054, did not 
reveal any meaningful trends. The possible exceptions were specific, isolated 
comments on low film supply, which, again, does not reveal any helpful informa- 
tion about the design of the panel or about the man/machine Interface, other than 
the suggestion that worry about resource depletion may have been a distraction. 

3. 2. 4. 2 SL-3 Procedures-Related Critical Incidents 

Half (117) the total 234 critical incidents scored on SL-3 involved procedu res . 
By referring to Figure 3-6, above, it can be seen that the scored cells under 
Schedule/lnstructions comprised the largest number of these procedural Incidents. 
Remarks about Operation Time Phasing were of next highest frequency, possibly 
Indicating that some of the errors scored under the Hardware category (e.g.. 
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inaccurate display setting) may ultimately be attributable to procedural problems 
(e.g., crewman being short on time, unsure of instructions and making best guesses). 

A summary of procedural critical incidents, rank ordered by experiment 
(and GEN, EPC) , is provided in Table 3-13. Many instances appear In the voice 
transcripts In which the crewmen attribute Incidents (scored in this report as 
hardware errors) to the quality of some procedures. This raises a question as to 
whether many difficulties with hardware were not so much an effect of design as 
of procedural directives. 

Scheduling/ Instructions 

About one-third of the remarks scored as errors under Schedule/lnstructions 
were noticed by the crewmen themselves. Most of their remarks under this category 
dealt with what were perceived as ambiguous or incomplete instructions and schedul- 
ing or instructional conflicts. Also, many of these remarks included more than one 
experiment or a whole ATM pass, which accounts, in part, for the high ranking of 
"General" procedural remarks in Table 3-13. 

The Experiment Pointing Control (EPC) had a fairly low ranking procedural ly, 
but this ranking may be misleading. Pointing Instruction problems, when related to 
specific experiments (such as S082B) appeared to contribute in a large degree to the 
rank ordering of those experiments. 

Experiment Operation 

Fifteen of the total SL-3 scored procedures critical incidents fell under the 
Experiment Operation category. Several were attributable to errors resulting from 
revised operational parameters. The largest number of remarks in this group dealt 
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Table 3-13: Rank Ordering of Procedural Critical Incidents on SL-3 


SCHEDULE/INSTRUCTIONAL 

EXPERIMENT 

OPERATION 

OPERATION 

TIME PHASING 

TOTAL 

COMMENTS 

S082B 

18 

S056 

3 

GEN 

9 

S082B 

29 

GEN 

16 

S082B 

3 

S082B 

8 

GEN 

28 

S054 

11 

GEN 

3 

S054 

5 

S054 

18 

S082A 

6 

S055 

2 

S055 

4 

S055 

1 1 

EPC 

6 

S054 

2 

S082A 

3 

S082A 

10 

S055 

5 

Ha 

1 

EPC 

3 

EPC 

9 

Y S056 

3 

S082A 

1 

S052 

3 

S056 

8 

5 Hcc 

0 

S052 

0 

S056 

2 

S052 

3 

S052 

_ 0 _ 

EPC 

_ 0 _ 

Ha 

_ 0 _ 

Ha 

_L 


65 


15 


37 


117 



specifically with experiment operation, although these remarks were distributed 
fairly equally across experiments. 

Time Phasing 

Of the 37 procedural incidents accrued under the Time Phasing category, half 
were errors noted by the crew due to beginning or terminating an experiment or 
Building Block out of phase with requested time. These remarks might, therefore, 
relate to incidents in the Schedule/lnstructions category. 

3.2.5 Skylab 4 Mission 

3.2. 5.1 SL-4 Hardware-Related Critical Incidents 

The SL-4 voice transcript data was approached in the same manner as the 
SL-3 data. Analysis of both missions was undertaken simultaneously after the 
analysts had developed and verified the analysis technique. 

Figure 3-7 presents the summarized voice transcript data for the SL-4 
mission. The chart is identical in format to the previous figure for SL-3 (Figure 
3-6) . The SL-4 mission was 42% longer in duration than SL-3, and man-hours 
of ATM operation on SL-4 exceeded those on SL-3 by 70%. Both of these factors, 
as would be expected, had an effect on the total number of remarks. In addition, 
the SL-4 crew appeared to verbalize more on ATM operations. But in spite of 
the larger number and greater density of remarks made by the SL-4 crew, they 
did not offer the depth of comment on hardware difficulties offered by the SL-3 
crew. This is explainable in light of the fact that the hardware problems were 
well known and documented by the SL-3 crew prior to involvement of the SL-4 


crew . 
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Table 3-14 presents the ranking of experiments by total number of deduced 
errors and total comments. As in the SL-3 Mission, the ranking based on comments 
does not differ substantially from the ranking based on deduced errors. 

The discussions which follow describe these errors and comments, and 
suggest possible causes and relationships between reported phenomena. 

S055 

The S055 experiment on SL-4, as on SL-3, had the highest number of 
critical incidents. Also, the SL-4 detector switches, as on SL-3, contributed the 
highest percentage, 39%, of these incidents. This likeness across missions 
becomes even more evident when the SL-4 crew's high number of S055 omission 
errors is compared with SL-3 S055 omission errors. In both cases, the prepon- 
derence of these errors occurred early in the flight, e.g., for SL~4, 84% of such 
errors occurred on three isolated days in the first 15 days of the mission. The 
major difference between the two missions, with respect to detectors. Is the absence 
in SL-4 of the Detector #5 malfunction mentioned on SL-3. 

Unlike SL-3, the SL-4 crew mentioned a higher degree of concern about 
the Grating Position Indicator. This hardware contributed 32% of total critical 
incidents on this experiment. Over half of the remarks regarding this piece of 
hardware indicated that the crewmen would pass by the desired grating position 
and have to either settle for an approximate setting or completely recycle the 
instrument. The problem arose out of the mechanical properties of the display; 
it counted upward, and could not be cycled backward. Consequently, obtaining 
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Table 3-14: Hardware Critical Incidents 
Rank-Ordered by Experiment 


75% 


25% 

TOTAL 

TOTAL 

ERRORS 

COMMENTS 

S055 

42 

EPC 12 

S052 

18 

S055 11 

S056 

15 

Ha 3 

S054 

11 

S082A 3 

S082B 

9 

S082B 3 

S082A 

4 

S052 3 

EPC 

4 

S054 2 

Ha 

3 

S056 0 

GEN 

0 

GEN 0 

106 

37 



100% 

TOTAL 

CRITICAL 

INCIDENTS 

S055 

53 

S052 

21 

EPC 

16 

S056 

15 

S054 

13 

S082B 

12 

S082A 

7 

H a 

6 

GEN 

0 


143 




a correct setting after a miss required approximately five minutes of additional 
time. 

The Intensity Data display received several comments relating the use of 
the instrument as a pointing aid requiring a relatively long learning period. 
However, the crew additionally commented that It was a useful tool. 

S052 

The S052 experiment is notable for its relatively high number of SL-4 
critical incidents, as compared to SL-3, These could not be related to any mean- 
ingful trends, however, due to their distribution over the entire SL-4 mission. 

The mode select, 5-posltion rotary switch received 10 of 21 of the total remarks, 
all of which were errors. Seven commission errors were attributable to mis- 
reading the Pad and three were Time Displacement errors due to the overextension 
of the Continuous mode. Of the total errors deduced In the voice transcript 
remarks, 23% were detected by ground support. Some comments reporting a 
"streak" or "bar" in the S052 display were made; these sometimes interfered with 
accurately locating certain features, such as the Comet Kohoutek, 

The Experiment Pointing Control, when mentioned by the SL-4 crew as 
a specific piece of hardware or mentioned in relation to the ATM experiments 
(Figure 3-7 under General Experiment Support Equipment) , frequently required 
more time than expected to achieve required operations, some of these problems 
may have arisen out of the schedule. Often the crew mentioned that pointing 
time Intefered with scheduled start times of JOPs and building blocks. Another 
difficulty commented on by the SL-4 crew which was not mentioned in SL-3 was 
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the tendency for the pointing to drift. This condition may have produced poor 
results on the experiments operating at that time. 

S056, S05^, SQ82A, SQ82B^ H-alpha, Hardware-Related Critical Incidents 

Hardware-related incidents were rather evenly distributed across the other 
experiments and over mission duration. Some few stand out from the' others, 
and these are discussed below. 

When looking at Figure 3-7, it may be seen that the Wavelength switch on 
S082B was reported to have had some errors. This component is a 3-position 
(M-F-M) toggle switch. Interestingly, no errors were reported on the similar 
component In S082A. S082A and B are the only experiments having timed modes 

which received scored critical incidents on the timer. S056, which received 
the highest total number of reported errors on the start/stop switch, had no 
other hardware comments recorded other than deduced errors. It was the third 
ranked experiment on SL-4 in total deduced errors recorded. The SL-4 crew . 
mentioned that the frames remaining counters were used as an indication of 
experiment status. 

3. 2.5.2 SL-4 Procedures-Related Critical Incidents 

The cell scores for procedural problems identified through voice transcript 
analysis are shown above in Figure 3-7. A ranking of experiments by type of 
procedure Is presented in Table 3-14. As in SL-3, the SL-4 procedures account 
for a high percentage, 42.5%, of the total SL-4 critical Incidents. 
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Table 3-15: Procedural Critical Incidents 

Rank-Order by Experiment for Total Mission 


Schedule/ Instructional 

Experiment Operation 

Operation Time 

Phasing 

Total Proc. 

Comments 

Gen 

12 

S055 

6 

S055 

16 

S055 

29 

S054 

9 

S082B 

4 

S082B 

9 

S082B 

16 

S055 

7 

S056 

2 

S054 

5 

S054 

16 

S056 

4 

S052 

2 

S082A 

4 

Gen 

15 

S082A 

3 

S054 

2 

S052 

4 

S056 

7 

S082B 

3 

Ha 

1 

EPC 

2 

S082A 

7 

EPC 

3 

EPC 

1 

Gen 

2 

S052 

6 

Ha 

2 

Gen 

1 

Ha 

1 

EPC 

6 

S052 

_0 

S082A 

0 

S056 

1 

Ha 

4 


43 


19 


44 


106 
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In comparing the ranking of experiments by procedural problems with 
the ranking by hardware remarks, substantial differences are uncovered. 

Although S055 continued to be the most discussed experiment, S082B, General 
Comments, EPC and S054 also received a high percentage of procedural remarks, 
especially In comparison to the low percent of hardware remarks scored for those 
categories. 

There Is a similar trend in SL-3. Most of the procedural critical incidents 
were directed toward scheduling of experiments and their instructions. The 
crew seeemed to mention most that the Instructions were confusing and unclear. 

In such areas as the Pad formatting, labeling and wording. Changes In procedures, 
"being engulfed with building blocks", having no observing time due to experi- 
ment overruns, and "tight" morning schedules were some of the procedural problems 
they mentioned. 

The SL-4 procedural critical incidents differed highly from SL-3 In the 
Operation Time Phasing area in the amount of comments made about ESS (Effective 
Sunset) termination. 54% of these procedural comments were on the termination 
of experiments at ESS, as compared to 8% in SL-3. 

The procedural difficulties on SL-4, In general, appear to be more related 
to time than the comments on SL-3. Building blocks were updated between the 
missions, a fact which may have eliminated some difficulties. However, the Inclusion 
of more activities, shopping lists, etc. was reported to have created more time- 
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lining and sequencing conflicts on SL-4. Observation periods were used as 
experiment periods on several occasions in SL-4. 


3.3 COMPARISONS 

,Two comparisons are made in this section: 1) a comparison of the results 

of the analyses of the two bodies of data considered during this study, and 2) a 
comparison of the results of the SL-2 ATM assessment study with the results of this 
study. 

3.3.1 Telemetry Data and Critical Incidents Data Comparison 

While both bodies of data analyzed contributed to the identification of ATM C&D 
problem areas, each data source was also able to contribute its own dimension to 
the study. For example, the statistical analysis of the telemetry data led to the 
identification and quantification of many factorial relationships. Those factorial 
relationships, in turn, led to the identification of problem areas for which design or 
procedural recommendations were formulated. 

The critical incidents data analysis led directly to the discovery of problem 
areas, and, while it did not allow any assessment in terms of absolute frequence of 
occurrence, it did provide circumstantial insight that aided in the formulation of 
appropriate design recommendations. Because of the nature of the two analytical 
techniques, it was unlikely that the results would stand in conflict. In fact, they 
did not. Rather, the results derived from the two bodies of data tend to be mutually 
supportive. For example, both techniques: 
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• uncovered problems in event timing. 

• uncovered procedural directives problems. 

• uncovered the, occurrence of deliberate procedural deviations. 

• uncovered many of the same design and use problems. 

• uncovered the same kinds of control -related problems. 

3.3.2 Comparison of Phase 2 and Phase 3 Results 

Of the seven SL-2 study results (Table 3-16), three (Nos. 1, 2, and 4) were 
in conflict with, two (Nos. 3 and 6) were in support of, and two (Nos, 5 and 7) 
were not comparable to the SL-3/SL-4 study results. 

Table 3-16: SL-2 ATM Assessment Study Results 

1} The left side of the panel had a higher deviation rate than the right side. 

2) Three-position toggle switches (F-F-F) used without a display had a high 

deviation rate (9%) . 

3) Experiment modes that required manual timing by the crewmen were often 
timed improperly. 

4) Many of the deviations which occurred during operation of S082B involved 
confusion with S082A. 

5) Building Blocks with several subsections separated by pointing commands 
had high deviation rates for the centermost subsections. 

6) Isolated control actuations had a much higher deviation rate (21%) than 
sequential control actuations (2%) . 

7) One-g control panel operation is a valid predictor of zero-g operation. 
Although the deviation rates are different, the types and relative frequency 
of deviations are almost Identical. 
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Specific comparisons between these SL-2 study results and the SL-3/SL-4 study 
results are described in Section 4,0. 
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SECTION 4.0 
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INTEGRATED RESULTS, CONCLUSIONS 
AND DESIGN RECOMMENDATIONS 

This section has three major objectives: 

1) to present the study results discussed in Section 3.0 as direct, abbre- 
viated statements. 

2) to conclude what factors contributed to the problem areas identified. 

3) to advance design and procedural recommendations for overcoming those 
factors. 

In meeting these objectives, the 52 study results obtained from this study are 
presented in nine tables, together with the number of the Section 3.0 paragraph 
from which they were extracted. Immediately following each table is a discussion 
of the factors contributing to the results found and the associated design or 
procedural remedies. 


4.1 INTER-MISSION RESULTS 

Table 4~1: Inter-Mission Results 


REFERENCE 

PARAGRAPH 


RESULT 


3. 1 . 1 .1 


3. 1 .2.2.3 
3. 1 . 1 . 1 


SL-4 overall error rate was about half that of SL-3. 

SL-3 error rate was significantly higher (<.001) than was that 
of SL-4. 

Both controls and displays contributed to the SL-3, SL-4 difference. 
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Table 4-1: Inter-Mission Results (Continued) 


REFERENCE 

PARAGRAPH 


RESULT 


3. 1 . 1 . 1 


S054 Exposure Range error rate was dramatically lower for SL-4 


than for SL-3. 


3. 1 .2.2.3 


Hal, S055A, S054 error rates were significantly lower for SL-4 
than for SL-3. 


All five of these study results can be attributed to the same four factors: 

1) differences in procedural directives, 2) differences in task difficulty, 3) differ 
ences in training effectiveness, and 4) differences in ground crew experience . 
While no effort was made to verify the correctness of these assumptions, it is 
assumed that because SL-4 followed two similar manned flights, it benefilted 
from previous experience. Therefore, It should be safe to say that, in addition 
to more experienced ground crews, SL-4 was provided with more understandable 
procedural directives, less confusing and better arranged tasks, and more effec- 
tive training. The design and procedural implications of these assumptions are 
clear; Where possible, prior to a mission, 

1) Develop an integrated set of well organized, clearly understandable 
operating procedures and procedural directives for both the crew 
and for direct ground support personnel. 

2) Train all crews and direct mission support personnel to use those 
procedures and procedural directives, using the highest possible 
fidelity integrated mission simulators. 

3) Where needed, as a result of the experiences gained in that training 
simulation, upgrade all procedures and procedural directives and 
train again. 
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Repeat Steps 1-3 until an acceptable level of overall system perform- 
ance has been attained, 

4,2 INTRA-MISSION RESULTS 

Table 4-2: Intra-Mission Results 


REFERENCE 

PARAGRAPH RESULTS 


3. 1.1.1 SL-3 error rate decreased across time. 


3. 1.1 . 1 


Display error rate decreased across time for both SL-3 and SL-4. 


3 . 1 . 2 . 2 . 3 


There were significant differences in error rates between time 
periods. 


3. 1.1.1 


SL-4 error rate remained constant over time (revised figures) , 


With the exception of the third item. Table H-2 results tend to confirm the 
recommendations advanced in Section 4.1. Whereas SL-4 mission performance 
tended to remain at a constant rate (thus tending to confirm suppositions that it 
received the benefit of more complete, effective training), SL-3 performance 
did not. Rather, SL-3 mission performance tended to improve over time. In doing 
so, it exhibited a learning/adaptation effect and, therefore, supports an argument 
for additional training. 

An interesting observation concerning the second result in Table 4-2 (e.g., 
both missions tended to improve in their display performances over time) is that 
neither mission crew had been sufficiently trained to use the displays provided 
for the tasks performed. Thus, recommendations to rectify problems associated 
with this section's results are the same as the Section 4.1 recommendations. 
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4.3 COMPONENT FUNCTION RELATED RESULTS 

Table 4-3: Component Function Related Results 

REFERENCE 

PARAGRAPH RESULT 

3. 1.1. 2 Operations Involving 3-pos F-F-F toggle switches (mode select 

function) were performed less accurately than those involving 3-pos 
M-SL-M toggle switches (predominantly Start/Stop function) . 

3. 1.1. 3. 4 Mode selection errors accounted for the fourth highest number of 
Table 3-1 error rate cells. 

3. 1.2.2. 1 Hypothesis #1* - Rejected 

3. 1.2. 2.1 Hypothesis #2** - Rejected 

3. 1.2. 2.1 Hypothesis #3*** - Rejected 

In this study, no instance of hardware type superiority over another hard- 
ware type was demonstrated. What did emerge from this study is that component 
function and associated relationships (e.g., panel placement, task sequence order, 
etc.) seemed to be more important. This stands in direct conflict with the findings 
of the SL-2 ATM assessment study results, which attributed high error rates to 

■^Hypothesis #1: Three-position toggle switches have a significantly higher error 

rate than other types of switches. 

**Hypothesis #2: Rotary switches with positions numbering more than four have 

a significantly higher error rate than rotary switches with four positions. 

***Hypothesis #3: Rotary switches with positions numbering more than four have 

a significantly higher error rate than the other types of switches used on the 
ATM panel. 
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the use of three position (F-F-F) toggle switches. Design implications of the 
Table 4-3 results are not obvious. Therefore, certain physical and procedural 
similarities along functional lines may help to provide useful insight. 

First, the Start/Stop function is always associated with a single control . 

Each of these is located more or less in the same position with respect to its 
associated experiments. Moreover, each is accompanied by a ready/operate 
light which enhances its target value. Together, these physical attributes and 
the prominence of start/stop instructions in the procedural directives would seem 
to be sufficient reasons for lower error rates. 

The mode select function, on the other hand, is served by a varying number 
of controls on different experiments. These controls are not located similarly 
within their respective experiments. The status displays associated with these con- 
trols are of relatively low target value. Further, they need to provide more than simple 

I 

"yes“ or '•no*' information. By themselves, these attributes would seem to be cause 
enough for degraded mode selection performance. However, in addition to these 
physical factors, the mode select instructions have been given a position of low prominence 
in the procedural directives. 

While It may seem appropriate to recommend that mode select functions be per- 
formed by one control, that the control be located consistently with respect to its 
function, and, that it be accompanied by a high target value display, these are often 
not the best design choices. However, one absolute procedural recommendation that . 
can be made as a result of the Table 4-3 findings, is that any procedural instruction 
to be communicated must be given sufficient prominence in the procedural directives. 
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4.4 CONTROL RELATED RESULTS 


Table 4-4: Control Related Results 


REFERENCE 

PARAGRAPH 

RESULT 

3.2. 1 . 1 

S055A HV detectors were omitted. 

3. 2. 1 . 1 

S055A grating reference (opt./mech) selector was left in wrong 
position. 

3. 2. 1.1 

S055A night interlock was left in override (experiment ran all 
night) . 

3. 2. 1.1 

S056 exposure selection switch was associated with commission 
errors (3-pos., F-F-F toggle switch). 

3.2.2. 1 

S052 mode selection caused errors (5-pos. rotary switch) . 

3.2.2. 1 

S054 grating position selection was left in improper position 
(2-pos., F-F toggle switch). 

3.2.2. 1 

S082B wavelength selection resulted in errors (3-pos. , M-SL-M 
toggle switch) . 


Rather than having been concluded from error rates calculated from tele- 
metry data as were the results in previous Section 4.0 tables, the results in 
Table 4-4 were extracted from the voice transcripts. Because of this, no assump- 
tions can be made about their relative contribution to the overall ATM C&D error 
rates. Nevertheless, since they were of enough concern to have been emphatically 
voiced by the crew or by first line mission support personnel, they will be addressed 
here. 

Of immediate Interest Is the diversity of the control population. This tends 
to support the results of Section 4.3. Also, since all the controls mentioned in 



Table 4-4 are "components", they are subject to the same recommendations advanced 
in Section 4.2. Other recommendations are presented below. 

There are seven S055A high voltage detectors (3~pos., F-F-F toggles) 
located together on the panel. While problems with operating these detectors did 
not appear in the telemetry analysis, failures to activate those that were required 
can probably be attributed to the combined effects of three major factors; 

1) directive information background, 2) individual control feedback, and 3) con- 
trol clutter. 


Directive Information breakdown Is the problem addressed at the end of 
Section 4.3. It has equal applicability to all of the controls listed in Table 4-4, 
Individual control feedback (visual) Is also applicable to all of the controls in 
this section, but its solutions are well understood. Therefore, except in instances 
where special comment is deemed advantageous, neither of these two factors will 
be mentioned again in this section. Control clutter, on the other hand, is a 
problem unique (among the results In Table 4-4) to the S055A high voltage detec- 
tors. It results from the inability to readily distinguish one control from another. 
The problem can be handled by rearrangement and re-marking, but the design 
recommendation applicable to control clutter is that similar proximal components 
should be made Individually distinguishable. 

S055A grating reference and S054 grating position status information were 
not always included among the procedural directives. Thus, there was no 
consistent reminder for the operator to check the related controls. Yet several 
Instances were recorded of valuable data being missed becaus^ these controls 
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were left in the wrong positions. 

This indicates that there was a need for the operator to check the status 
of each of these controls on a regular basis. Where this need exists, it must 
be reflected by inclusion of appropriate instructions as part of the operational 
directives . 

Leaving the S055A night interlock in the override position was another 
problem which arose. While the problem is similar to the one just discussed, 
there is an important difference. Whereas the grating reference control can be 
considered a normal part of the experiment, the night interlock switch cannot. 

The latter is a control rarely used and, then, under special conditions, so 
instructions concerning its positional status do not rightfully belong in the normal 
operational directives. Therefore, if It Is inadvertently left "on”, it is unlikely 
that the error will be quickly discovered. To guard against the possibility of 
controls of this category , being inappropriately left on, a cyclical alert (preferably 
tonal) should be Incorporated. That is, if a control that can adversely affect system 
performance is left in a dangerous position beyond an acceptable time period, an 
Intermittent tone should be sounded to alert the operator of its status. 

The S056 exposure length switch is unusual only in that it is isolated from 
the rest of its associated components. If possible, isolation of this type should 
be avoided. If it cannot, particular care must be taken to include necessary 
instructions prominently In the pertinent operational directives. 

The problem cited for the S052 mode select switch specifically mentioned 
it as being attributable to a misread pad (i.e., procedural directives). Therefore, 
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the procedural directives comment already made should suffice here. 

Comments about the S082B wavelength selectors may also have been rooted 
in problems with procedural directives, but. In this case, there is an additional 
problem. The control (M-SL-M toggle switch) itself gives no feedback, so a three- 
position flag is provided for that purpose. If there was a problem with mis-settlng this 
control, it well may have been due to a display legibility problem. Therefore, 
if the need for display feedback supplementary to a control action has been iden- 
tified, the designer must ascertain that the required information is faithfully 
transmitted to the control user under all expected operating conditions (e.g., low 
light levels) . 


4.5 Ha 1 FRAMES/MINUTE CONTROL RELATED RESULTS 

Table 4-5: Ha 1 Frames/Min Control Related Results 


REFERENCE 

PARAGRAPH 

RESULT 

3.1 , 1 .3.3 

H« 1 frames/min. rate selection errors accounted for the third 
highest number of Table 3-1 error rate cells. 

3. 1 .2.2.2 

Hypothesis #5* accepted (<.001). 

3. 1 .2.2.2 

Left side panel operations demonstrated significantly higher (<.0 
error rates than did right side operations; this effect vanishes if 
the Hal errors are removed. 


(Hypothesis #5; Hal has a significantly higher error rate than any other 
experiment on the left side of the panel. 
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Table 4-5; H«l Frames/Min Control Related Results (Continued) 


REFERENCE 

PARAGRAPH 

RESULT 

3. 1 .2.2.2 

Highly similar experiments exhibited significantly higher error 
rates than dissimilar experiments (this result is attributable to 
Ha 1 error rate skewing effect) . 

3. 1 .2.2.2 

There is no significant difference in error rates between S082A 
and S082B. 

1 

3.2. 1 . 1 

Ground control discovered Hal frames/min. setting errors. 


While it may seem unusual or inappropriate to devote an entire section 
to the findings surrounding one experiment, the impact of this experiment on 
overall ATM C&D performance measured, justifies the special treatment. First, 
Ha 1 frames/min. error rates alone accounted for the first four results listed in 
Table 4-5. Thus, if the reasons for the high Hoc 1 error rates can be specified, 
the mechanism behind these results will also have been pinpointed. 

Section 3.1 . 1 .1 attributes aJ_[of the Hoc errors for SL-4 to an inadequacy of 
this study (missing data). This accounts for much of the problem. In fact, 
it also supports the assertion that the remainder of the problems experienced 
were not due to hardware inadequacies. Rather, it indicates that the trouble 
may have had two other sources: training emphasis and procedural directives. 

In the first case, it Is known that the importance of H oc was minimized 
throughout many of the training sessions (because of its relative operating 
simplicity and its lack of full experiment stature) . Indeed, it is also apparent 
that this minimization of stature was carried through all the way to the Ha l 
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operational directives presented in the Building Blocks, e.g., as tiny notations 
buried in obscure rows at the bottom of the form. Thus, both sources seem to 
have been contributory to the frames/min. error rate problem. 

If a similar problem Is to be avoided in future missions, it is imperative 

that: 

1) All mission operations be given appropriate stature during training. 

2) All required procedures be presented In the procedural directives with 
equal prominence. 

) 

Having discussed the reasons behind the large Ha error rates, it is now 
appropriate to turn to a discussion of the Table 4-5 results. First, it must be 
said that, originally. Results #3 and #4 tended to support the SL-2 ATM assess- 
ment's left vs. right side and S082A vs S082B results. As amended by the removal 
of Ha 1 errors, however, the results are in disagreement. So, too, is this 
study's finding comparing S082A and S082B error rates (Result #4) in opposition 
to the SL-2 assessment finding addressing the same question. 

4.6 DISPLAY RELATED RESULTS 

Table 4-6: Display Related Results 


REFERENCE 


PARAGRAPH 

RESULT 

3 . 1 . 1 . 1 

Both crews performed 3 to 5 times better on controls than they 


did on displays. 

3.1 .2.2. 1 

Display operations resulted in significantly higher (<.001) error 


rates than did control operations. 
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Table 4-6; Display Related Results (Continued) 


REFERENCE 

PARAGRAPH RESULT 

3. 1.1. 3, 2 S055A grating position errors were the second highest contributor 

to the number of Table 3-1 error rate cells. 


3. 1.2. 2. 2 Sequential operations exhibited significantly ( <.001) lower 
error rates than did single operations. 

3. 2. 2.1 S055A grating position and indicator contributed to operating diffi- 

culties . 


3. 2. 1.1 

3. 2. 1.1 

3.2. 1 .1 

3 . 2 . 2 . 1 


H a video monitor and images were of poor quality (rapid image 
oscillations and telescoping effect) . 

Ha video image and quality created problems operating S082A. 

3-position flags for S056 filler position proved difficult to use. 

Experiment pointing would drift off required position undetected 
by operator. 


Result #4 in the above table is directly supportive of the similar, isolated 
control actuations finding reported as a result of the SL-2 assessment activity. 
Further, both of those findings are more or less supportive of Table 4-6 results 
#1 and #2, That is because display operations are largely equivalent to isolated 
or single operations. They are not, however, exactly similar. The S055A 
grating position indicator, usually used as part of a sequential operation, is 
the exception. Nevertheless, its performance, like that of the other displays, 
left a great deal to be desired. 
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The detailed reasons for the disappointing performance of the ATM displays 
studied was presented earlier In Section 3. 1.1.1. For this reason, only a general 
overview of the problems encountered will be identified. 

All of the first five results can be explained in terms of the effects of three 
factors: directive functional breakdown, display target value, and display dynamics 
Results ^?1 , #2 and #4 had directive functional breakdowns and display target value 
problems related to timing to account for their presence (discussed in Section 4.7 
below) . Timing, in turn, was dependent upon effective procedural directives and 
compelling display target values. The S055A grating position indicator results, on 
the other hand, from display problem relationships (requirements to dyanmics) . 
Specifically, the procedural directives required setting the display units digit to a 
particular value. That requirement was not compatible with the display's high rate 
of incrementing. Thus, an overrun situation often arose, which was compounded by 
the lack of a display decrement provision. At least three design recommendations 
for eliminating this problem are apparent: 

1) Where a digital counter (indicative of a non-continuous control require- 
ment equipment function) must be set, a pre-select feature should be 
provided. That is, the desired display value should be ordered by the 
operator and the equipment should effect the setting. 

2) If there is a need for precisely setting a digital counter display, and, if 
no provision can be made for its counting direction reversal, then a single 
rate of advancement well suited to the task requirements should be 
provided . 

3) If a high counting rate digital display is required, a variable rate control 
with a reversing feature should be provided. 
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Both Hal video monitoring comments pertain to the same problem - poor 
video quality due to rapid image oscillations and a telescoping effect. These problems 
can be eliminated if the following CRT monitor design criteria are met: 

1) Scaling disparity between expected or reference and primary images 
should be minimized. 

2) Sufficient image resolution should be provided to permit operator acquisi- 
tion of required data. 

3) (mage pattern and pattern orientation should conform to operator expecta- 
tions. 

4) Sufficient image stability must be provided to assure operator acquisition 
of the information presented. 

The three-position flags that indicate the S056 filter position presented the 
same problem discussed about the S082B wavelength select display in Section 4.4. 
That is, it was misread. Thus, the discussion presented in the earlier section also 
pertains here. 

Experiment pointing drift as mentioned in the last comment is another matter. 

It can lead to a serious impairment of mission objectives across many experiments. 
Therefore, if encountering such a problem is likely or even probable, a suitable 
alarm system should be provided to indicate when the monitored function drifts out- 
side tolerance . 
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4.7 TIME DISPLACEMENT RELATED RESULTS 


Table 4-7: Time Displacement Related Results 

REFERENCE 

PARAGRAPH RESULT 

Time displacement errors accounted for the highest number of Table 3-1 
error rate cells, 

S052 continuous mode was allowed to run past time. 

Timed operations information in unclear. 

Problems associated with time displacement fell Into two categories: procedural 
directives and functioning. Together, these problems caused the late starting and 
early termination of experiments that required manual timing. This directly supports 
the SL-2 ATM assessment finding that "experiment modes that require manual timing 
by the crewmen were often timed improperly." 

Specific procedural directive problems related to time displacement errors 
included; misunderstood voice communications, cluttered checklists, and inade- 
quate Building Block presentation of experiment start, stop and span times. These 
suggest the following recommendations; 

IJ Vocally communicated timing directives should be presented in a format, 
mutually acceptable to both sender and recipient. That format should be 
capable of transmission (via the intended medium) without significant loss 
of information Intelligibility. 

2) Checklists should be laid out to ensure full and accurate transmission of 
the information they contain. 


3.1 . 1 .3. 1 

3.2. 1 . 1 

3.2. 1 .2 
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3) A list of manually performed activities should have the point of initiation 
for all activities clearly indicated along a single timeline. In addition, 
ail of the operations important to the performance of each activity should 
be clearly presented and time referenced. 

Part of the display functioning problems associated with time displacement were 
attributable to a lack of target value*. This was true both for clocks (digital and 
analog) and for ready/operate lights (the two display types used for timing] . Because 
both of these displays are dependent upon the visual channel for their target value, 
and because the visual channel is often not available, the solution is obvious. If 
manually timed operations are required, the time reference display should be equipped 
with an auditory alarm. 

The remainder of the display problem was attributable to the lack of a dedicated 
timing device for each experiment, incorporating manually timed operating modes. 
Here, too, the design solution is obvious — all manually timed operations likely to 
be performed in conjunction with other manually timed operations, should be provided 
with dedicated timers. 


Target value the ability to attract attention when competing with other stimuli. 
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4.8 PROCEDURES/SCHEDULING RELATED RESULTS 


Table 4-8: Procedures and Scheduling Related Results 

REFERENCE 

PARAGRAPH RESULT 

Both crews erred in following assigned procedures. 

Pointing instructions were unclear as to coordinate and permissible 
tolerance. 

Important information was omitted from procedural directives {per- 
taining to the S055A Night Interlock, and the Crating Reference (opt,/ 
mech) . 

Nearly all S082B and S054 comments cited procedural ambiguity. 

S082B scheduling difficulties were experienced. 

Insufficient time was scheduled for experiment pointing. 

Each of the first four Table 4-8 results can be attributed to factors that have 
already been identified and fully addressed. Therefore, neither the results nor the 
factors to which they are attributed will receive the benefit of further discussion. 

Both of the remaining Table 4-8 results address a scheduling problem. 

From these and other related voice transcript remarks, it would seem that realistic 
time requirements had not been developed prior to the mission for each of the planned 
tasks . 

If the day to day mission planning is to be scheduled effectively, such time 
estimates must be provided. It is, therefore., recommended that during the training 
simulations identified In Section 4.1, an effort be made to accurately record the lime 
required to perform ^planned mission tasks. 


3. 1 . 1 . 1 

3.2.1 .2 

3.2.1 .2 

3.2. 1 .2 

3.2. 2.2 

3.2.2. 1 
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4.9 UNRELATED FINDINGS AND COMMENTS 

Table 4-9; Unrelated Findings and Comments 


REFERENCE 

PARAGRAPH 

RESULT 

3. 1.2. 2. 2 

There Is no significant difference between upper and lower control 
panel associated error rates. 

3.2. 1 . 1 

S055A Grating Reference select control (2-posltlon, F-F toggle) was 
confused with the S055A start/stop switch (3-positlon, M-SL-M 
toggle switch) . 

3.2.2. 1 

Pointing by S055A intensity data display was effective but required 
practice. 

3.2.2. 1 

Frames remaining counters were used to indicate operating status. 

3. 2.2.2 

Dislike was expressed for the use of two control mode select functions 
(e.g., S054) . 


Because these results are more or less unrelated to each other, they will be 
addressed one at a time. 

Finding no significant error rate differences between upper and lower panel 
operations complements the no right-left differences picture presented in Section 4.5. 
Taken together, these results more or less indicate that, within the ATM panel area 

studies, all segments are equally suitable to the control and display operations per- 
formed . 

Mistaking the S055A grating reference control for the start/stop switch probably 
resulted because the two components were adjacent and both were under illuminated 
displays. Thus, each of the components had an equal target value that was higher 
than that of those surrounding them. Since it is usually the case that only the 
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start/stop switch has the advantage of high target value, both components were 
perceived as the start/stop switch. Therefore, if a control Is intended to have high 
target value, that target value should not be diminished by placing a similar control, 
equally able to attract the operator's attention, next to it. 

The third comment concerning the need to practice experiment pointing using 
the S055A intensity data display only underlines the requirement for the high fidelity 
training simulations identified in Section 4. 1 . 

Use of the frames remaining counters, as described in Comment #4, to indicate 
an experiment operating status is not a negative finding. It only points out that the 
operators want to know more than that they have started the experiment or that they 
have selected the correct mode. They also want to know if the mode they have selected 
is being performed. Therefore, in situations where such information is not already 
provided, it is desirable to provide the operator with the feedback necessary to tell 
him in what mode the system Is actually operating. 

Finally, the last comment was included to demonstrate that crew subjective 
comments, while extremely valuable, are not necessarily related to actual system 
performance. In this case, the comment was made during SL-4 indicating a dislike 
for the dual control mode select operations required for S054. From that comment, 
one might expect that an analysis of the telemetry data would reveal relatively high 
error rates related to S054 control operations. This was not the case. In fact, during 
the SL-4 Mission, S054 scored the lowest (.005) control-related error rate of all the 
ATM experiments. Thus, while the voice transcript comments were treated throughout 
Section 4.0 as though they were indicative of a serious C&D system problem, in many 
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cases, they probably were not. Nevertheless, problems that were only perceived 
served the same purpose as the "real" ones, i.e., they focused analytical attention on 
particular areas of the ATM experiment C&D system operation and, in doing so, 
triggered the development of design guidelines that might be of some future benefit.^ 


4.10 DESIGN RECOMMENDATIONS SUMMARY 

Here, in Table 4-10 is a list of all of the design and procedural recommendations 
advanced in the Section 4.0 subsections. While reviewing this list, there may be a 
tendency on the part of the reader to consider some of the solutions obvious. That 
the solutions seem obvious after being stated does not minimize their importance; 
they do address problem areas identified as a result of a careful scrutiny of SL-3 and 
SL-4 ATM CSD system performance. 

Table 4-10: Design Recommendations Summary 


1) Develop an integrated set of well organized, clearly understandable operating 
procedures and procedural directives, both for the crew and for direct ground 
support personnel. 

2) Train all crews and direct mission support personnel to use the procedures and 
procedural directives developed, using the highest possible integrated mission 
simulations . 

3) Where needed, as a result of the experiences gained in training simulation, 
upgrade all procedures and procedural directives and train again. 

4) Repeat steps 1 through 3 until an acceptable level of overall system performance 
has been attained. 

5) Any procedural instruction to be communicated must be given sufficient promi- 
nence in the procedural directives. 
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Table 4-10: Design Recommendations Summary 

(Continued) 


6} Similar proximal components should be made individually distinguishable. 

7) Where there is a need for the operator to check control setting status, the need 
must be reflected by the inclusion of appropriate Instructions as part of the 
operational directives. 

8) If a control that can adversely affect system performance is left in a dangerous 
position beyond acceptable time limits, an intermittent tone should be sounded 
to alert the operator of Its status. 

9) If the need for display feedback supplementary to a control action has been 
identified, the required information must be faithfully transmitted to the control 
user under all expected operating conditions (e.g., low light levels) . 

10) All mission operations should be given appropriate stature during training, 

11) All required procedures should be presented in the procedural directives with 
equal prominence. 

12) Where a digital counter (indicative of a noncontinuous control requirement 
equipment function) must be set, a preselect feature should be provided. 

That is, the desired display value should be ordered by the operator and the 
equipment should effect the setting, 

13) If there is a need for precisely setting a digital counter display, and if no 
provision can be made for its counting direction reversal, then a single rate 
of advancement well suited to the task requirement should be provided. 

14) If a high counting rate digital display is required, a variable rate control with 
a reversing feature should be provided. 

15) Scaling disparity between expected or reference and primary CRT images 
should be minimized. 

16) Sufficient CRT image resolution should be provided to permit operator acquisi- 
tion of required data. 

17) CRT image pattern and pattern orientation should conform to operator expecta- 
tions. 
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Table 4-10; Design Recommendations Summary 
(Continued) 


18) Sufficient CRT image stability must be provided to assure operator acquisition 
of the information presented. 

19) If important display parameter drift is likely or even probable, a suitable 
alarm system should be provided to indicate when the monitored function 
drifts outside tolerance. 

20) Vocally communicated timing directives should be presented in a format mutually 
acceptable to both sender and recipient. That format should be capable of 
transmission (via the intended medium) without significant loss of information 
intelligibility. 

21) Checklists should be laid out to ensure full and accurate transmission of the 
information they contain. 

22) A list of manually performed operations should have the point of initiation for 
all activities clearly indicated along a single timeline. In addition, all of the 
operations important to the performance of each activity should be clearly 
presented and time referenced . 

23) If manually timed operations are required, the time reference display (clock) 
should be equipped with an auditory alarm. 

24) All manually timed operations likely to be performed in conjunction with other 
manually timed operations should be provided dedicated timers. 

25) The time required to perform ^ planned mission tasks should be accurately 
measured and recorded during high fidelity training sessions. 

26) If a control (or display) is intended to have a high target value, that target 
value should not be diminished by placing a similar control (or display) , 
equally able to attract the operator's attention, next to It. 

27) It is desirable to provide the operator with the feedback necessary to tell him 
in what mode the system is actually operating. 
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SECTION 5.0 



RECOMMENDATIONS FOR FUTURE WORK 

Because of the scope of this effort, much valuable work was left undone. 

For example, many interactions between the Table 3~1 (above) data cell conti'ibutors 
were not identified. Further, many individual voice transcript Incidents were not 
addressed. Performing these activities would be profitable in terms of triggering 
additional design recommendations. 

More important contributions could be made, however, if the data collected 
and the results obtained were correlated with other identifiable performance impacting 
factors. Some of these factors might include work density (as inferred from general 
mission and AtM schedules), crew physiological status (as indicated by biomedical 
data), and incidents of hardware failure (from the Skylab mission log) . 

A final effort that could be of significant benefit Is the preparation of a sample 
set of SL-3 or SL-4 procedural directives that incorporate the Section 4.0 recommenda'- 
tions. The recommendations of this study, together with a set of the present directives 
and a set of the revised directives for comparison, could contribute to the effective- 
ness of future mission procedural directives. 


5-1 


APPENDIX A 


LIST OF REFERENCES 



APPENDIX A 


REFERENCES 


An Evaluation of the ATM Man/Machine Interface (SL-2 Data Reduction and Analysis), 
Phase 2 Report. URS/Matrix Company, Man Systems Division, Huntsville, 
Alabama, 1974. 


Skylab Mission Events (SL-1/2, SL-3, and SL-4) , 25M00700, George C. Marshall 
Space Flight Center, Huntsville, Alabama, 1974. 


Program for Accounting the ATM Schedule (PAAS) , Final Summary Report, Parametric 
Studies of ATM Experiment Operations, Contract NAS8-24009 ED-2002- 1 722, 
March 29, 1974. 


Edwards, A. L. Experimental Design in Psychological Research . New York: 
Holt, Rinehart and Winston, 1962. 


Runyon, R. P., and A. Haber. Fundamentals of Behavioral Statistics . Reading, 
Mass.: Addison-Wesley , 1971. 


Winer, B. J. Statistical Principals in Experimental Design . New York: McGraw- 

Hill Co. , 1962. 



appendix b 


TELEMETRY DATA SUMMARY SHEETS 

I 


RAW DATA SCORE SHEET 


DATA POINT #: — (/| JOP-#: 5*^-' 3 B8- #; £- 79' MISSION TIME: 


[o ■D'ly 2?3 


CONTROLS 


^ y y ^ 


DISPLAYS 


CREW OPN / ERROR TYPE 








r 

\ 

% 

1 

\ 



V 



\ 

~ 

\ 



1 


\ 

_ 





\ 


\ 




DATA POINT TOTAL 


KEY; 

H ERRORS COMMITTED 

ERROR OPPORTUNITIES 


■ - 3 - I 

' t ‘ ~ 


P<tet4*^«v •■ 'll 

PHOtecAk/^*! P«.v>»WejJ •. II) 
I ’ Wf 



til 


Ti?tfiL, PRXeeOORAL o*v. » 26 



Figure B-la; Data Summary Sheet for First 1/3 of SL-3 
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Figure B- Ic: Data Summary Sheet for Third 1/3 of SL-3 


luiiiui BU 





















RAW DATA SCORE SHEET 


DATA POINT #: 


mission TIME: Disv jp JS'O 

^ CONTROLS ”7 displays /cREW OPN /error TYPE 




remarks 




*? / Av 

fA 







- 

- 

— — 



- 

- 

1 

»VocA6AL»ra t>&/i44i«o:i 

■X 

- 

- 

- 

- 





9'ee 


f^nxjteJunc htyl'eV iot» ■ I 



DATA POINT TOTAL 


KEY: 

H ERRORS COMMITTED 

ERROR OPPORTUNITIES 




74\ 


Y V V Y \ 

wa1a‘X4>a4I ' ^ 


Total PRoCE-Etiuet C*.VIRTIOnS ■» 


Figure B Id; Data Summary Sheet for First 1/3 of SL-4 
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Figure B-le: Data Summary Sheet for Second 1/3 of SL-4 
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Figure B-2a: Whole Mission Data Summary Sheet for SL-3 













RAW DATA SCORE SHEET 




TOTF^U 

DATA POINT #: 

JOP #; 

BB #: 

MISSION TIME: 


CREW OPN X ERROR TYPE 


/ / / x^vvX-v / / / 


DATA POINT TOTAL 




3. 


3 - 5 * 


6 5 --^ 


JO M'S 


"RwcafliiLOTs' E f-fon — tj 


ERRORS COMMITTED 
ERROR OPPORTUNITIES 


Figure B-2b: Whole Mission Data Summary Sheet for SL~4 





























